caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

caracal related singularity images #1527

Closed Chole98 closed 1 month ago

Chole98 commented 1 year ago

The server I used already intalled Singularity. But there are notes on it "Please note that the compute nodes on Lengau do not have access to the internet, which means that users will not be able to use the Singularity pull command. However, users can use the scp.chpc.ac.za node to facilitate the transfer of images to and from the platform."

Considering this restriction, I would appreciate guidance regarding the necessary images to download and their respective locations for using CARACal effectively.

Could you kindly provide me with information on which specific images I need to obtain for CARACal and where I can access them?

Chole98 commented 1 year ago

I tried to stimela pull or singularity pull, but didn't work. export SINGULARITY_PULLFOLDER=/mnt/lustre/users/mli/datareduction/singularity_images stimela pull -s traceback stimela.utils.StimelaCabRuntimeError: cd . && singularity returns error code 255 How could I access to /software/astro/caracal/STIMELA_IMAGES_1.7.0 in manual download way rather than singularity pull/stimela pull? then I believe the problem will be solved.

Athanaseus commented 1 year ago

@Chole98, sorry for not getting back to you in time.

Can you please try: ... -ct singularity -sid /software/astro/caracal/STIMELA_IMAGES_1.7.0

The Stimela version used supposed to match the images (/software/astro/caracal/STIMELA_IMAGES_1.7.0) intended to be used; otherwise, new/old images might attempt to download.

PS: For stimela pull -s, check if setting the environment variable STIMELA_PULLFOLDER works.

Could you kindly provide me with information on which specific images I need to obtain for CARACal and where I can access them?

Currently, this can be achieved automatically when running CARACal. i.e. it will download the relevant image as it runs.

I hope these help,

kind regards.

Chole98 commented 12 months ago

@Athanaseus Thanks for your help. I had installed stimela version 1.7.8 and set the environment variableSTIMELA_PULLFOLDER, but also returned stimela.utils.StimelaCabRuntimeError: cd . && singularity returns error code 255. I believe the reason stimela pull -s failed is the same as singularity pull failed, it's because as I said the compute nodes do not have access to the internet, which means that users will not be able to use the Singularity pull command. So I have to download the corresponding singularity images first(in manual) and then transfer them to that computer. But I don't know where I could download these singularity images. Any links?

Athanaseus commented 10 months ago

Hi @Chole98, the reason for using -sid /software/astro/caracal/STIMELA_IMAGES_1.7.6 is to ensure that stimela-1.7.6 use pre build images in the existing folder '/software/astro/caracal/STIMELA_IMAGES_1.7.6'. You won't need to download anything in this instance. If the version of stimela you intend to use differs from the version number on this folder -sid, downloading will likely occur.

If you want to download manually, you can run singularity pull on a node with internet and copy the images to the singularity folder you want to work with.

Most of the images are hosted on quay.io

e.g. singularity pull --name stimela_casa_1.7.1.sif docker://quay.io/stimela/casa:1.7.1

Chole98 commented 1 month ago

Thanks,I appreciate it