search

sylabs / singularity

SingularityCE is the Community Edition of Singularity, an open source container platform designed to be simple, fast, and secure.
https://sylabs.io/docs/
Other
759 stars 97 forks source link

could not attach image file to loop device: no loop devices available #67

Closed a-ludi closed 3 years ago

a-ludi commented 3 years ago

I recently encountered the following error with Singularity 3.7.3 on Arch linux. It used to work for me but now fails:

$ singularity pull --name hello.simg shub://vsoch/hello-world
$ singularity run hello.simg
FATAL:   container creation failed: mount /proc/self/fd/3->/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

There are certainly enough loop devices available:

$ ls /dev/loop*
/dev/loop0    /dev/loop121  /dev/loop145  /dev/loop169  /dev/loop192  /dev/loop215
(and more...)
$ mount | grep loop
(empty output)

The config contains max loop devices = 256.

Here is the debug output for the same command:

$ singularity -d run hello.simg
DEBUG   [U=1000,P=681364]  persistentPreRun()            Singularity version: 3.7.3
DEBUG   [U=1000,P=681364]  persistentPreRun()            Parsing configuration file /etc/singularity/singularity.conf
DEBUG   [U=1000,P=681364]  handleConfDir()               /home/alu/.singularity already exists. Not creating.
DEBUG   [U=1000,P=681364]  execStarter()                 Saving umask 0022 for propagation into container
DEBUG   [U=1000,P=681364]  execStarter()                 Checking for encrypted system partition
DEBUG   [U=1000,P=681364]  Init()                        Image format detection
DEBUG   [U=1000,P=681364]  Init()                        Check for sandbox image format
DEBUG   [U=1000,P=681364]  Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=1000,P=681364]  Init()                        Check for sif image format
DEBUG   [U=1000,P=681364]  Init()                        sif format initializer returned: SIF magic not found
DEBUG   [U=1000,P=681364]  Init()                        Check for squashfs image format
DEBUG   [U=1000,P=681364]  Init()                        squashfs image format detected
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding SHELL environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding SESSION_MANAGER environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding COLORTERM environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LESS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_SESSION_PATH environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_MENU_PREFIX environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding CONDA_EXE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding _CE_M environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding HISTSIZE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding SSH_AUTH_SOCK environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XMODIFIERS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding DESKTOP_SESSION environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LC_MONETARY environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding EDITOR environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding DBUS_STARTER_BUS_TYPE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding PWD environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LOGNAME environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_SESSION_DESKTOP environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding QT_QPA_PLATFORMTHEME environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_SESSION_TYPE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding CONDA_PREFIX environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding MAMBA_ROOT_PREFIX environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding MANPATH environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding SYSTEMD_EXEC_PID environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XAUTHORITY environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_GREETER_DATA_DIR environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding MOTD_SHOWN environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding GDM_LANG environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LC_PAPER environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LANG environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LS_COLORS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_CURRENT_DESKTOP environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding DARWIN environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding VTE_VERSION environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding CONDA_PROMPT_MODIFIER environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_SEAT_PATH environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding MPW_FULLNAME environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding INVOCATION_ID environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding MANAGERPID environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding INFOPATH environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding NVM_DIR environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_SESSION_CLASS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding TERM environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding _CE_CONDA environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding USER environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LIBRARY_PATH environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding MICRO_MYOTIS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding CONDA_SHLVL environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding DISPLAY environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding SHLVL environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding NVM_CD_FLAGS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding PBGAPS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding PAGER environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding QT_IM_MODULE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LC_MEASUREMENT environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding DBUS_STARTER_ADDRESS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding OOO_FORCE_DESKTOP environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding COOKIE_VERIFICATION_SECRET environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding DENTIST environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding CONDA_PYTHON_EXE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding TILIX_ID environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding XDG_RUNTIME_DIR environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding CONDA_DEFAULT_ENV environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LC_TIME environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding JOURNAL_STREAM environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LC_COLLATE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding GTK3_MODULES environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding GDMSESSION environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding HISTFILESIZE environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding DBUS_SESSION_BUS_ADDRESS environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding HG environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding NVM_BIN environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding MAIL environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding LC_NUMERIC environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding OLDPWD environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding _ environment variable
DEBUG   [U=1000,P=681364]  SetContainerEnv()             Forwarding USER_PATH environment variable
VERBOSE [U=1000,P=681364]  SetContainerEnv()             Setting HOME=/home/alu
VERBOSE [U=1000,P=681364]  SetContainerEnv()             Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG   [U=1000,P=681364]  init()                        Use starter binary /usr/lib/singularity/bin/starter-suid
VERBOSE [U=0,P=681364]     print()                       Set messagelevel to: 5
VERBOSE [U=0,P=681364]     init()                        Starter initialization
DEBUG   [U=0,P=681364]     load_overlay_module()         Trying to load overlay kernel module
DEBUG   [U=0,P=681364]     load_overlay_module()         Overlay seems supported by the kernel
VERBOSE [U=0,P=681364]     is_suid()                     Check if we are running as setuid
VERBOSE [U=0,P=681364]     priv_drop()                   Drop root privileges
DEBUG   [U=1000,P=681364]  read_engine_config()          Read engine configuration
DEBUG   [U=1000,P=681364]  init()                        Wait completion of stage1
VERBOSE [U=1000,P=681373]  priv_drop()                   Drop root privileges permanently
DEBUG   [U=1000,P=681373]  set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1000,P=681373]  init()                        Spawn stage 1
DEBUG   [U=1000,P=681373]  startup()                     singularity runtime engine selected
VERBOSE [U=1000,P=681373]  startup()                     Execute stage 1
DEBUG   [U=1000,P=681373]  StageOne()                    Entering stage 1
DEBUG   [U=1000,P=681373]  prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point
DEBUG   [U=1000,P=681373]  prepareAutofs()               Could not keep file descriptor for bind path /etc/localtime: no mount point
DEBUG   [U=1000,P=681373]  prepareAutofs()               Could not keep file descriptor for bind path /etc/hosts: no mount point
DEBUG   [U=1000,P=681373]  prepareAutofs()               Could not keep file descriptor for home directory /home/alu: no mount point
DEBUG   [U=1000,P=681373]  prepareAutofs()               Could not keep file descriptor for current working directory /home/alu/projects/pb-gaps/src/dentist-example: no mount point
DEBUG   [U=1000,P=681373]  Init()                        Image format detection
DEBUG   [U=1000,P=681373]  Init()                        Check for sandbox image format
DEBUG   [U=1000,P=681373]  Init()                        sandbox format initializer returned: not a directory image
DEBUG   [U=1000,P=681373]  Init()                        Check for sif image format
DEBUG   [U=1000,P=681373]  Init()                        sif format initializer returned: SIF magic not found
DEBUG   [U=1000,P=681373]  Init()                        Check for squashfs image format
DEBUG   [U=1000,P=681373]  Init()                        squashfs image format detected
DEBUG   [U=1000,P=681373]  setSessionLayer()             Overlay seems supported and allowed by kernel
DEBUG   [U=1000,P=681373]  setSessionLayer()             Attempting to use overlayfs (enable overlay = try)
VERBOSE [U=1000,P=681364]  wait_child()                  stage 1 exited with status 0
DEBUG   [U=1000,P=681364]  cleanup_fd()                  Close file descriptor 4
DEBUG   [U=1000,P=681364]  cleanup_fd()                  Close file descriptor 5
DEBUG   [U=1000,P=681364]  cleanup_fd()                  Close file descriptor 6
DEBUG   [U=1000,P=681364]  init()                        Set child signal mask
DEBUG   [U=1000,P=681364]  init()                        Create socketpair for master communication channel
DEBUG   [U=1000,P=681364]  init()                        Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=1000,P=681364]  priv_escalate()               Get root privileges
VERBOSE [U=0,P=681364]     priv_escalate()               Change filesystem uid to 1000
VERBOSE [U=0,P=681364]     init()                        Spawn master process
DEBUG   [U=0,P=681379]     set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=0,P=681379]     create_namespace()            Create mount namespace
VERBOSE [U=0,P=681364]     enter_namespace()             Entering in mount namespace
DEBUG   [U=0,P=681364]     enter_namespace()             Opening namespace file ns/mnt
VERBOSE [U=0,P=681379]     create_namespace()            Create mount namespace
DEBUG   [U=0,P=681364]     set_master_privileges()       Set master privileges
DEBUG   [U=0,P=681364]     apply_privileges()            Effective capabilities:   0x00000000000000c0
DEBUG   [U=0,P=681364]     apply_privileges()            Permitted capabilities:   0x000001ffffffffff
DEBUG   [U=0,P=681364]     apply_privileges()            Bounding capabilities:    0x000001ffffffffff
DEBUG   [U=0,P=681364]     apply_privileges()            Inheritable capabilities: 0x000001ffffffffff
DEBUG   [U=0,P=681364]     apply_privileges()            Ambient capabilities:     0x0000000000000000
DEBUG   [U=0,P=681364]     apply_privileges()            Set user ID to 1000
DEBUG   [U=0,P=681380]     set_rpc_privileges()          Set RPC privileges
DEBUG   [U=0,P=681380]     apply_privileges()            Effective capabilities:   0x0000000000200000
DEBUG   [U=0,P=681380]     apply_privileges()            Permitted capabilities:   0x000001ffffffffff
DEBUG   [U=0,P=681380]     apply_privileges()            Bounding capabilities:    0x0000000008204000
DEBUG   [U=0,P=681380]     apply_privileges()            Inheritable capabilities: 0x0000000000000000
DEBUG   [U=0,P=681380]     apply_privileges()            Ambient capabilities:     0x0000000000000000
DEBUG   [U=0,P=681380]     apply_privileges()            Set user ID to 1000
DEBUG   [U=1000,P=681380]  set_parent_death_signal()     Set parent death signal to 9
VERBOSE [U=1000,P=681380]  init()                        Spawn RPC server
DEBUG   [U=1000,P=681364]  startup()                     singularity runtime engine selected
VERBOSE [U=1000,P=681364]  startup()                     Execute master process
DEBUG   [U=1000,P=681380]  startup()                     singularity runtime engine selected
VERBOSE [U=1000,P=681380]  startup()                     Serve RPC requests
DEBUG   [U=1000,P=681364]  setupSessionLayout()          Using Layer system: overlay
DEBUG   [U=1000,P=681364]  setupOverlayLayout()          Creating overlay SESSIONDIR layout
DEBUG   [U=1000,P=681364]  addRootfsMount()              Mount rootfs in read-only mode
DEBUG   [U=1000,P=681364]  addRootfsMount()              Image type is 4096
DEBUG   [U=1000,P=681364]  addRootfsMount()              Mounting block [squashfs] image: /home/alu/projects/pb-gaps/src/dentist-example/hello.simg
DEBUG   [U=1000,P=681364]  addKernelMount()              Checking configuration file for 'mount proc'
DEBUG   [U=1000,P=681364]  addKernelMount()              Adding proc to mount list
VERBOSE [U=1000,P=681364]  addKernelMount()              Default mount: /proc:/proc
DEBUG   [U=1000,P=681364]  addKernelMount()              Checking configuration file for 'mount sys'
DEBUG   [U=1000,P=681364]  addKernelMount()              Adding sysfs to mount list
VERBOSE [U=1000,P=681364]  addKernelMount()              Default mount: /sys:/sys
DEBUG   [U=1000,P=681364]  addDevMount()                 Checking configuration file for 'mount dev'
DEBUG   [U=1000,P=681364]  addDevMount()                 Adding dev to mount list
VERBOSE [U=1000,P=681364]  addDevMount()                 Default mount: /dev:/dev
DEBUG   [U=1000,P=681364]  addHostMount()                Not mounting host file systems per configuration
VERBOSE [U=1000,P=681364]  addBindsMount()               Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=1000,P=681364]  addBindsMount()               Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=1000,P=681364]  addHomeStagingDir()           Staging home directory (/home/alu) at /var/singularity/mnt/session/home/alu
DEBUG   [U=1000,P=681364]  addHomeMount()                Adding home directory mount [/var/singularity/mnt/session/home/alu:/home/alu] to list using layer: overlay
DEBUG   [U=1000,P=681364]  addTmpMount()                 Checking for 'mount tmp' in configuration file
VERBOSE [U=1000,P=681364]  addTmpMount()                 Default mount: /tmp:/tmp
VERBOSE [U=1000,P=681364]  addTmpMount()                 Default mount: /var/tmp:/var/tmp
DEBUG   [U=1000,P=681364]  addScratchMount()             Not mounting scratch directory: Not requested
DEBUG   [U=1000,P=681364]  addLibsMount()                Checking for 'user bind control' in configuration file
DEBUG   [U=1000,P=681364]  addFilesMount()               Checking for 'user bind control' in configuration file
DEBUG   [U=1000,P=681364]  addResolvConfMount()          Adding /etc/resolv.conf to mount list
VERBOSE [U=1000,P=681364]  addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG   [U=1000,P=681364]  addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG   [U=1000,P=681364]  create()                      Mount all
DEBUG   [U=1000,P=681364]  mountGeneric()                Mounting tmpfs to /var/singularity/mnt/session
FATAL   [U=1000,P=681364]  Master()                      container creation failed: mount /proc/self/fd/3->/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

@dtrudg What to you mean by mountinfo? Here is the output of mount before launching the command:

proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=3914504k,nr_inodes=978626,mode=755,inode64)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755,inode64)
/dev/mapper/AntergosVG-AntergosRoot on / type ext4 (rw,noatime,discard)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11543)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
/dev/sda1 on /boot type ext4 (rw,noatime,discard,stripe=4)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,nr_inodes=409600,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=785652k,nr_inodes=196413,mode=700,uid=1000,gid=100,inode64)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=100)
/dev/mmcblk0p1 on /run/media/alu/0BF9-003D type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2)

Originally posted by @a-ludi in https://github.com/sylabs/singularity/issues/65#issuecomment-851313653

dtrudg commented 3 years ago

Thanks for the information there - it's not yet clear exactly what is going on.

When you say it used to work, but now doesn't - did 3.7.3 work previously, or was it an older version that works?

Has your kernel recently been upgraded? Could you give output of uname -a just in case that's related? Also check in dmesg output for any errors that might look relevant. Thanks.

dtrudg commented 3 years ago

@alfaro96 - thanks for this information. That environment will likely be a bit easier for us to replicate than the exact arch Linux one.

I'll be able to take a deeper look into this tomorrow.

alfaro96 commented 3 years ago

@alfaro96 - thanks for this information. That environment will likely be a bit easier for us to replicate than the exact arch Linux one.

I'll be able to take a deeper look into this tomorrow.

Thank you @dtrudg for the quick response!

I have granted access to the loop devices and the problem has been solved. That is why I deleted the original comment, to avoid disturbance in this issue.

a-ludi commented 3 years ago

Sadly, I do not know exactly which version worked for me and I cannot try it out because the release tarballs on GitHub are not available anymore - it likely was v3.5.2.

Yes, my kernel has been updated. It is continuously being updated, that's part of Arch linux.

There are no message in dmesg when I try to run the container.

Here is the system info:

$ uname -a
Linux *** 5.12.3-arch1-1 #1 SMP PREEMPT Wed, 12 May 2021 17:54:18 +0000 x86_64 GNU/Linux
dtrudg commented 3 years ago

Hi @a-ludi - I haven't been able to replicate yet, in a simple arch VM I have. This is going to need a bit more effort when I have an afternoon, as we don't routinely test on Arch Linux, and I cannot reproduce under Fedora which I regularly use and is tracking new kernels.

Sadly, I do not know exactly which version worked for me and I cannot try it out because the release tarballs on GitHub are not available anymore - it likely was v3.5.2.

Please see: https://github.com/sylabs/singularity/issues/68

The older releases can be found at github.com/hpcng/singularity/releases

This location is a fork from SingularityCE 3.8.0 onward. Unfortunately it was previously a redirect to the hpcng/singularity address and many were not aware of the prior move to that hpcng/singularity location.

a-ludi commented 3 years ago

Quick update: 3.5.2 works. (Thanks for the pointer to hpcng for some reason I did not find it there earlier.)

I will try to determine the latest working version and report back.

a-ludi commented 3 years ago

Weirdly, after trying different version, 3.7.3 now works just fine - the error is gone. Maybe the error arose because build files from older release were kept and reused for the 3.7.3 build. Thanks for your support anyway!

dtrudg commented 3 years ago

Thanks for updating us!

richelbilderbeek commented 1 year ago

@alfaro96 I have exactly the same problem and you seem to have found a solution. I quote:

I have granted access to the loop devices and the problem has been solved.

How did you do this?

Would be great to find en even better solution here :-)

hiiiyilingzhang commented 1 year ago

I've encountered the issue after reboot server. It worked totally fine for v3.10.2 before reboot, but it shows the following errors now: 

FATAL:   container creation failed: mount /proc/self/fd/3->/usr/local/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available

Then I update to v3.11.4 since I notice v3.11.2 patch release fix the bug by using /dev/loop-control for loop device creation. But it issue still exits.

Same with the issue author, there are enough loop devices available:

$ ls /dev/loop*
/dev/loop0    /dev/loop121  /dev/loop145  /dev/loop169  /dev/loop192  /dev/loop255  (and more...)
$ mount | grep loop
(empty output)

Here's system info:

uname -a
Linux xxx-ubuntu 5.15.0-78-generic #85~20.04.1-Ubuntu SMP Mon Jul 17 09:42:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Looking forward to your replies.

dtrudg commented 1 year ago

If you have an issue with loop device creation, that is not solved by updating to the current version of SingularityCE, then please open a new issue for it - providing as much information as you can. It is likely to have a different cause than old issues reported with older versions of Singularity and the kernel.