search

ratt-ru / Stimela-classic

Containerized radio interferometry scripting framework -- NB: Classic version is no longer in active development, use stimela 2! See README for details.
GNU General Public License v2.0
28 stars 16 forks source link

stimela pull -s error on ILIFU #620

Open o-smirnov opened 4 years ago

o-smirnov commented 4 years ago

Anyone seen this?

# 2020/05/14 17:01:48  warn xattr{var/lib/apt/lists/auxfiles} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
# 2020/05/14 17:01:48  warn xattr{/scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/var/lib/apt/lists/auxfiles} destination filesystem does not support xat
trs, further warnings will be suppressed                                              
# 2020/05/14 17:01:50  info unpack layer: sha256:d972307d9fa8968bb7b272efbb88bdbf55ca0c31c4e580d2e626e946edf95ac0
# 2020/05/14 17:01:51  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
# 2020/05/14 17:01:51  warn xattr{/scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/etc/gshadow} destination filesystem does not support xattrs, further wa
rnings will be suppressed                                                             
# 2020/05/14 17:04:20  info unpack layer: sha256:b428246021ce6e0d64dfe248fb2510810ae75eeedc28159a9d306f4bdb605218
# 2020/05/14 17:04:21  warn xattr{etc/gshadow} ignoring ENOTSUP on setxattr "user.rootlesscontainers"
# 2020/05/14 17:04:21  warn xattr{/scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/etc/gshadow} destination filesystem does not support xattrs, further wa
rnings will be suppressed                                                             
# FATAL:   While performing build: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: usr/lib/python3.7/distutils/README: link: unpriv.li
nk: unpriv.wrap linkname: link /scratch/users/oms/singularity_tmp/rootfs-099f2131-95f3-11ea-aab7-fa163e13d453/usr/lib/python3.6/distutils/README /scratch/users/oms/singularity_tmp/root
fs-099f2131-95f3-11ea-aab7-fa163e13d453/usr/lib/python3.7/distutils/README: operation not permitted
Traceback (most recent call last):                                                    
  File "/users/oms/.venv/caracal/bin/stimela", line 7, in <module>                    
    exec(compile(f.read(), __file__, 'exec'))                                         
  File "/users/oms/projects/Stimela/bin/stimela", line 14, in <module>                
    main.main([a for a in sys.argv[1:]])                                              
  File "/users/oms/projects/Stimela/stimela/main.py", line 592, in main               
    _cmd(argv)                                                                        
  File "/users/oms/projects/Stimela/stimela/main.py", line 309, in pull               
    image, simage, directory=pull_folder, force=args.force)                           
  File "/users/oms/projects/Stimela/stimela/singularity.py", line 43, in pull         
    os.path.join(directory,store_path), fp])                                            
  File "/users/oms/projects/Stimela/stimela/utils/xrun_poll.py", line 189, in xrun    
    raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: singularity returns error code 255

My folders are set properly:

$  set | grep SINGU
SINGULARITY_CACHEDIR=/scratch/users/oms/singularity_cache
SINGULARITY_PULLFOLDER=/scratch/users/oms/singularity_images
SINGULARITY_TMPDIR=/scratch/users/oms/singularity_tmp
KshitijT commented 4 years ago

@o-smirnov , which image is this for? I see the weird messages too, but my stimela pull (albeit the one by running CARACal) seems to have gone through. E.g. for owlcat:

# 2020/05/13 22:14:20  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/root} destination filesyste
m does not support xattrs, further warnings will be suppressed
# 2020/05/13 22:14:25  info unpack layer: sha256:cc429581b6e11b942ddbe730b10902f5bcd06da5b3a6112f08fa05f7f1657e3b
# 2020/05/13 22:14:25  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/root} ignoring ENOTSUP on c
learxattrs
# 2020/05/13 22:14:25  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/root} destination filesyste
m does not support xattrs, further warnings will be suppressed
# 2020/05/13 22:15:19  info unpack layer: sha256:0bf0e60a13745d325c320c845f0d37ab72e5d1de832b94644240743d15a21aec
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/usr} ignoring ENOTSUP on cl
earxattrs
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/usr} destination filesystem
 does not support xattrs, further warnings will be suppressed
# 2020/05/13 22:15:19  info unpack layer: sha256:95dc7348703240c1de73f765b098b559c62b49169c3b9fd7eac9bab1aeca3a7f
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/etc} ignoring ENOTSUP on cl
earxattrs
# 2020/05/13 22:15:19  warn xattr{/idia/users/kshitij/.singularity_tmp/rootfs-0a1c9f7e-9556-11ea-9790-fa163eefae2a/etc} destination filesystem
 does not support xattrs, further warnings will be suppressed
# INFO:    Creating SIF file...
o-smirnov commented 4 years ago

For all of them... but maybe it depends on filesystem, let me check...

HaoChen-ast commented 4 years ago

Another error in pull Stimela image on ilifu. But as we have Stimela_image on ilifu now at /idia/software/containers/STIMELA_IMAGES https://github.com/caracal-pipeline/caracal/issues/1125, we can use that STIMELA_IMAGES directly (no need to pull again).

# 2020/05/20 09:12:43  info unpack layer: sha256:5a424948411eaa63e57cc6f817865b444b32a5f130c57b5bd7bccdc2e3e03b61
# INFO:    Creating SIF file...
# FATAL:   While performing build: while creating squashfs: create command failed: exit status 1:
# Write failed because No space left on device
# 
# FATAL ERROR:Failed to write to output filesystem
Traceback (most recent call last):
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/bin/stimela", line 14, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/main.py", line 593, in main
    _cmd(argv)
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/main.py", line 310, in pull
    image, simage, directory=pull_folder, force=args.force)
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/singularity.py", line 49, in pull
    image_path, fp])
  File "/idia/projects/vela/CARACal_all/CARACal-18May2020/CARACal-venv-18May2020/lib/python3.7/site-packages/stimela/utils/xrun_poll.py", line 189, in xrun
    raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: singularity returns error code 255
KshitijT commented 4 years ago

@HaoChen-ast , this is a slightly different error.

Could you please set the following (as per Jordan's suggestion) in your bashrc file:

export SINGULARITY_CACHEDIR=/software/astro/caracal/.singularity_cache
export SINGULARITY_TMPDIR=/software/astro/caracal/.singularity_tmp

Should do the trick.

HaoChen-ast commented 4 years ago 
export SINGULARITY_CACHEDIR=/software/astro/caracal/.singularity_cache
export SINGULARITY_TMPDIR=/software/astro/caracal/.singularity_tmp

There is no directory of /software/astro/caracal/ on ilifu.

/idia/software/containers/STIMELA_IMAGES/ is enough for us. No need to pull by ourselves again. Please remember to keep it updating on time. Thanks, @KshitijT!

SpheMakh commented 4 years ago

Has this been resolved?

ulricharmel commented 4 years ago

I am getting this error on tina when I run stimela pull -s I don't know if it is related to this issue

Cleaning up... ERROR: Failed squashing image, left template directory at: /tmp/.singularity-build.XrN9KQ ERROR: pulling container failed! Traceback (most recent call last): File "/home/ulrich/stimela_venv/stimela3/bin/stimela", line 14, in main.main([a for a in sys.argv[1:]]) File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/main.py", line 602, in main _cmd(argv) File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/main.py", line 319, in pull image, simage, directory=pull_folder, force=args.force) File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/singularity.py", line 46, in pull name, fp]) File "/home/ulrich/stimela_venv/stimela3/lib/python3.6/site-packages/stimela/utils/xrun_poll.py", line 189, in xrun raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status)) stimela.utils.StimelaCabRuntimeError: cd . && singularity returns error code 1

INgauriS commented 1 year ago

export SINGULARITY_CACHEDIR=/software/astro/caracal/.singularity_cache export SINGULARITY_TMPDIR=/software/astro/caracal/.singularity_tmp

I did updated my bashrc for above line with uptodate stimela path: /idia/software/containers/STIMELA_IMAGES "export SINGULARITY_CACHEDIR=/idia/software/containers/STIMELA_IMAGE/.singularity_cache export SINGULARITY_TMPDIR=/idia/software/containers/STIMELA_IMAGES/.singularity_tmp"

I still have problem in pulling stimela (after 3 years) :(, How to resolve this. Copying the log file line below:

2023-04-13 11:06:55 CARACal.Stimela.obsconf INFO: Adding cab '/idia/software/containers/STIMELA_IMAGES/stimela_casa_1.7.1.sif' (5.8.0) to recipe, container name 'listobs-ms0' 2023-04-13 11:06:55 CARACal.Stimela.obsconf INFO: Adding cab '/idia/software/containers/STIMELA_IMAGES/stimela_msutils_1.4.6.sif' (1.0.1) to recipe, container name 'summary_json-ms0'

running cd /idia/software/containers/STIMELA_IMAGES/ && singularity pull --name stimela_owlcat_1.6.6.sif docker://quay.io/stimela/owlcat:1.6.6

WARNING: Cache disabled - cache location /idia/software/containers is not writable.

INFO: Converting OCI blobs to SIF format

FATAL: While making image from oci registry: error fetching image to cache: while building SIF from layers: unable to create new build: failed to create build parent dir: stat /idia/software/containers/STIMELA_IMAGES/.singularity_tmp: no such file or directory

2023-04-13 11:06:58 CARACal ERROR: cd /idia/software/containers/STIMELA_IMAGES/ && singularity returns error code 255 [StimelaCabRuntimeError] 2023-04-13 11:06:58 CARACal INFO: More information can be found in the logfile at output/logs-20230413-110655/log-caracal.txt 2023-04-13 11:06:58 CARACal INFO: You are running version 1.0.6 2023-04-13 11:06:58 CARACal ERROR: Traceback (most recent call last): 2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/caracal/main.py", line 187, in __run 2023-04-13 11:06:58 CARACal ERROR: pipeline.run_workers() 2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/caracal/workers/worker_administrator.py", line 441, in run_workers 2023-04-13 11:06:58 CARACal ERROR: worker.worker(self, recipe, config) 2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/caracal/workers/obsconf_worker.py", line 95, in worker 2023-04-13 11:06:58 CARACal ERROR: recipe.add("cab/owlcat_plotelev", step, { 2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/recipe.py", line 612, in add 2023-04-13 11:06:58 CARACal ERROR: job.setup_job(image=image, config=config, 2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/recipe.py", line 298, in setup_job 2023-04-13 11:06:58 CARACal ERROR: singularity.pull(image_url, 2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/singularity.py", line 44, in pull 2023-04-13 11:06:58 CARACal ERROR: utils.xrun(f"cd {directory} && singularity", ["pull", 2023-04-13 11:06:58 CARACal ERROR: File "/users/gsharma/.local/lib/python3.8/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun 2023-04-13 11:06:58 CARACal ERROR: raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status)) 2023-04-13 11:06:58 CARACal ERROR: stimela.utils.StimelaCabRuntimeError: cd /idia/software/containers/STIMELA_IMAGES/ && singularity returns error code 255 2023-04-13 11:06:58 CARACal INFO: exiting with error code 1

INgauriS commented 1 year ago

Problem resolved!

Required:

Stable of version of Caracal, on date 13/04/2023: caracal version 1.0.6 along with this /software/astro/caracal/STIMELA_IMAGES_1.7.5 Note: if working on ILIFU then do not* use /idia/software/containers/STIMELA_IMAGES/ , instead go with /software/astro/caracal/STIMELA_IMAGES_x.x.x

PeterKamphuis commented 1 year ago

So I'm getting Oleg's original error on our GlowHPC so it would be great to know if there is a solution for this. This is with singularity 3.4.2-1:

 Copying config sha256:ba3e353764d42bc1cd3d23ca9f3b7baa423baaac0eece17e426e80a108a009d5
# 
#  0 B / 6.33 KiB [--------------------------------------------------------------]
#  6.33 KiB / 6.33 KiB [======================================================] 0s
# Writing manifest to image destination
# Storing signatures
# 2023/07/04 16:12:40  info unpack layer: sha256:5b7339215d1d5f8e68622d584a224f60339f5bef41dbd74330d081e912f0cddd
# FATAL:   While making image from oci registry: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: bin: apply hdr metadata: clear xattr metadata: /home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/sbuild-171818620/fs/bin: unpriv.lclearxattrs: unpriv.lremovexattr: operation not supported
Traceback (most recent call last):
  File "/home/pkamphuis/venv/caracal_latest_venv/bin/stimela", line 11, in <module>
    main.main([a for a in sys.argv[1:]])
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/main.py", line 409, in main
    _cmd(argv)
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/main.py", line 314, in pull
    singularity.pull(
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/singularity.py", line 44, in pull
    utils.xrun(f"cd {directory} && singularity", ["pull", 
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/pkamphuis/venv/caracal_latest_venv/lib/python3.11/site-packages/stimela/utils/xrun_poll.py", line 227, in xrun
    raise StimelaCabRuntimeError("{} returns error code {}".format(command_name, status))
stimela.utils.StimelaCabRuntimeError: cd . && singularity returns error code 255
PeterKamphuis commented 1 year ago

Updating to singularity-ce version 3.9.9 changes the error slightly.

 2023/07/05 09:34:37  info unpack layer: sha256:5b7339215d1d5f8e68622d584a224f60339f5bef41dbd74330d081e912f0cddd
# 2023/07/05 09:34:37  warn xattr{/home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/build-temp-4126336251/rootfs/bin} ignoring ENOTSUP on clearxattrs
# 2023/07/05 09:34:37  warn xattr{/home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/build-temp-4126336251/rootfs/bin} destination filesystem does not support xattrs, further warnings will be suppressed
# FATAL:   While making image from oci registry: error fetching image to cache: while building SIF from layers: packer failed to pack: while unpacking tmpfs: error unpacking rootfs: unpack layer: unpack entry: bin: restore parent directory: restore xattr metadata: /home/pkamphuis/singularity_images/caracal_images/.singularity_tmp/build-temp-4126336251/rootfs: unpriv.lsetxattr: invalid argument

Can this be related to the file system being Ceph. Installing on a normal ubuntu system with singularity version 3.8.6 seems to go through correctly.