search

caracal-pipeline / caracal

Containerized Automated Radio Astronomy Calibration (CARACal) pipeline
GNU General Public License v2.0
28 stars 6 forks source link

Failed to create user namespace: user namespace disabled #1567

Closed arpan-52 closed 5 months ago

arpan-52 commented 5 months ago

Let me first describe the setup.

  1. I have made a python 3.8 virtual environment with Anaconda.
  2. Installed singularity with conda, version 3.8.6
  3. Installed Stimela version 1.7.9 with Conda.
  4. I have also specified the path for the singularity cache and temp directories.
  5. After pulling all the images with singularity I tried to run my .yml file.
  6. Whatever I am trying to do I am getting an error..

The error.....

INFO: Converting SIF file to temporary sandbox...

FATAL: while extracting /data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif: root filesystem extraction failed: extract command failed: ERROR : Failed to create user namespace: user namespace disabled

: exit status 1

2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: /data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17060054084448693 --containall --userns returns error code 255 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: job failed at 2024-01-23 15:53:31.281294 after 0:00:01.924652 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: Traceback (most recent call last): 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: job.run_job() 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: self.job.run(output_wrangler=self.apply_output_wranglers) 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/singularity.py", line 128, in run 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: utils.xrun(f"{BINARY} run --workdir {self.execdir} --containall {extras}", 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status)) 2024-01-23 15:53:31 STIMELA.image_target_1 ERROR: stimela.utils.StimelaCabRuntimeError: /data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17060054084448693 --containall --userns returns error code 255 2024-01-23 15:53:

From the initial inspection, it looks to me as a singularity problem. Is not conda installed singularity sufficient? Do we need a singularity with root permissions?

KshitijT commented 5 months ago

This seems similar to #1532. @arpan-52 do you permissions to pull the singularity images on the machine you are using?

arpan-52 commented 5 months ago

Yes, the images are pulled successfully.

-rw-rw-r-- 1 yanda yanda 207 Dec 15 15:08 log-stimela.txt -rw-rw-r-- 1 yanda yanda 3.1K Dec 15 15:08 stimela.config.deps drwxrwxr-x 2 yanda yanda 4.0K Dec 15 16:17 STIMELA_IMAGES -rw-rw-r-- 1 yanda yanda 765 Dec 21 23:06 start.sh drwxrwxr-x 3 yanda yanda 27 Jan 23 15:48 singularity_cache drwxrwxr-x 10 yanda yanda 242 Jan 24 20:13 singularity_tmp drwxrwxr-x 5 yanda yanda 247 Jan 24 20:13 test_caracal

STIMELA_IMAGES are the folder with all the images. yanda is my user name.

KshitijT commented 5 months ago

Could you please specify your caracal run command and also append the full log?

arpan-52 commented 5 months ago

I have used

stimela run ddf_stimela_hogbom_mask.py

This .py file contains all the recipes to do the imaging.

arpan-52 commented 5 months ago

log:

2025-02-05 22:39:23 STIMELA.DE calibration INFO: --------------------------------- 2024-02-05 22:39:23 STIMELA.DE calibration INFO: Stimela version 1.7.9 2024-02-05 22:39:23 STIMELA.DE calibration INFO: Running: DE calibration 2024-02-05 22:39:23 STIMELA.DE calibration INFO: --------------------------------- 2024-02-05 22:39:23 STIMELA.DE calibration INFO: Config directory cannot be found. Will create /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17071529638081439/stimela_parameter_files 2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif' (x.x.x) to recipe, container name 'image_target_1' 2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_breizorro_1.7.4.sif' (0.1.0) to recipe, container name 'mask1' 2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif' (x.x.x) to recipe, container name 'image_target_2' 2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_cubical_1.7.8-dev.sif' (1.6.3) to recipe, container name 'calibrate_dd' 2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_breizorro_1.7.4.sif' (0.1.0) to recipe, container name 'mask2' 2024-02-05 22:39:24 STIMELA.DE calibration INFO: Adding cab '/data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif' (x.x.x) to recipe, container name 'image_target_3' 2024-02-05 22:39:24 STIMELA.DE calibration INFO: STEP 1 :: image_target_1::ddfacet 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Completed jobs : [] 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Remaining jobs : ['mask1', 'image_target_2', 'calibrate_dd', 'mask2', 'image_target_3'] 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: mask1:: Make initial mask 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: image_target_2::ddfacet 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: calibrate_dd::cubical 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: mask2:: Make initial mask 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Logging remaining task: image_target_3::ddfacet 2024-02-05 22:39:28 STIMELA.DE calibration INFO: Saving pipeline information in .last_DE_calibration.json

FATAL: while extracting /data/manisha/TEST_DATA/arpan/STIMELA_IMAGES/stimela_ddfacet_1.7.1.sif: root filesystem extraction failed: extract command failed: ERROR : Failed to create user namespace: user namespace disabled : exit status 1 /data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17071529638081439 --containall --userns returns error code 255 job failed at 2024-02-05 22:39:28.804534 after 0:00:03.951822 Traceback (most recent call last): File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 713, in run job.run_job() File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/recipe.py", line 425, in run_job self.job.run(output_wrangler=self.apply_output_wranglers) File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/singularity.py", line 128, in run utils.xrun(f"{BINARY} run --workdir {self.execdir} --containall {extras}", File "/data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status)) stimela.utils.StimelaCabRuntimeError: /data/manisha/TEST_DATA/arpan/anaconda3/envs/38data/bin/singularity run --workdir /data/manisha/TEST_DATA/arpan/test_caracal/.stimela_workdir-17071529638081439 --containall --userns returns error code 255

KshitijT commented 5 months ago

@arpan-52 this is a stimela issue, not a CARACal one, could you please close this issue and open it on the stimela repository?

arpan-52 commented 5 months ago

Hi, I finally managed to tackle the issue with a system-wide updated version of singularity installation. The issue is now gone and the whole code works well. Thanks a lot.