search

sylabs / singularity

SingularityCE is the Community Edition of Singularity, an open source container platform designed to be simple, fast, and secure.
https://sylabs.io/docs/
Other
767 stars 98 forks source link

could not attach image file to loop device: no loop devices available #2144

Open MariusCausemann opened 1 year ago

MariusCausemann commented 1 year ago

Version of Singularity singularity-ce version 3.11.4

Describe the bug When running singularity container in parallel (hybrid mode), the image creation fails with "no loop device available". This used to work before but fails since the machine got updated to a newer kernel and OS. Error message:

mpirun -n 4 singularity -v run shub://vsoch/hello-world
INFO:    Use cached image
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INFO:    Use cached image
INFO:    Use cached image
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INFO:    Use cached image
VERBOSE: Setting HOME=/home/mariusca
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Set messagelevel to: 4
VERBOSE: Starter initialization
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Check if we are running as setuid
VERBOSE: Drop root privileges
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Drop root privileges permanently
VERBOSE: Spawn stage 1
VERBOSE: Execute stage 1
VERBOSE: Execute stage 1
VERBOSE: stage 1 exited with status 0
VERBOSE: Get root privileges
VERBOSE: Change filesystem uid to 6494
VERBOSE: Spawn master process
VERBOSE: Create mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Create mount namespace
VERBOSE: Execute stage 1
VERBOSE: Execute stage 1
VERBOSE: Spawn RPC server
VERBOSE: stage 1 exited with status 0
VERBOSE: Get root privileges
VERBOSE: Change filesystem uid to 6494
VERBOSE: Spawn master process
VERBOSE: Create mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Create mount namespace
VERBOSE: Spawn RPC server
VERBOSE: stage 1 exited with status 0
VERBOSE: stage 1 exited with status 0
VERBOSE: Get root privileges
VERBOSE: Get root privileges
VERBOSE: Change filesystem uid to 6494
VERBOSE: Change filesystem uid to 6494
VERBOSE: Spawn master process
VERBOSE: Spawn master process
VERBOSE: Create mount namespace
VERBOSE: Create mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Entering in mount namespace
VERBOSE: Create mount namespace
VERBOSE: Create mount namespace
VERBOSE: Execute master process
VERBOSE: Spawn RPC server
VERBOSE: Spawn RPC server
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Serve RPC requests
VERBOSE: Serve RPC requests
VERBOSE: Execute master process
VERBOSE: Execute master process
VERBOSE: Execute master process
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Serve RPC requests
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Serve RPC requests
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Default mount: /proc:/proc
VERBOSE: Default mount: /sys:/sys
VERBOSE: Default mount: /dev:/dev
VERBOSE: Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE: Found 'bind path' = /etc/hosts, /etc/hosts
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Default mount: /tmp:/tmp
VERBOSE: Default mount: /var/tmp:/var/tmp
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf
VERBOSE: Checking for template passwd file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Creating passwd content
VERBOSE: Creating template passwd file and appending user data: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Default mount: /etc/passwd:/etc/passwd
VERBOSE: Checking for template group file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/group
VERBOSE: Creating group content
VERBOSE: Default mount: /etc/group:/etc/group
VERBOSE: /home/mariusca found within container
VERBOSE: rpc server exited with status 0
VERBOSE: Execute stage 2
FATAL:   container creation failed: mount /proc/self/fd/3->/cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available
VERBOSE: Checking for template passwd file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Creating passwd content
FATAL:   container creation failed: mount /proc/self/fd/3->/cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs error: while mounting image /proc/self/fd/3: failed to find loop device: could not attach image file to loop device: no loop devices available
VERBOSE: Creating template passwd file and appending user data: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE: Default mount: /etc/passwd:/etc/passwd
VERBOSE: Checking for template group file: /cm/shared/apps/singularity-ce/3.11.4/var/singularity/mnt/session/rootfs/etc/group
VERBOSE: Creating group content
VERBOSE: Default mount: /etc/group:/etc/group
--------------------------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
--------------------------------------------------------------------------
RaawwWWWWWRRRR!! Avocado!
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[63632,1],1]
  Exit code:    255
--------------------------------------------------------------------------

To Reproduce Steps to reproduce the behavior: Run e.g.:

mpirun -n 4 singularity -v run shub://vsoch/hello-world

Expected behavior I expect the container to run.

OS / Linux Distribution Which Linux distribution are you using?

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

Additional context

Maybe related to the kernel version: 5.15.0-79-generic

dtrudg commented 1 year ago

Workaround here: https://github.com/sylabs/singularity/issues/1499#issuecomment-1493908443

See also an Ubuntu bug related to this: https://bugs.launchpad.net/ubuntu/+source/linux-hwe-5.15/+bug/2013086

dtrudg commented 1 year ago

We should document this formally, and perhaps show an error that mentions max_loop as a pointer.

Note that the kernel patch that changes the behaviour only appears to affect Ubuntu, as it sets CONFIG_BLK_DEV_LOOP_MIN_COUNT=8 in its kernel config, while others set CONFIG_BLK_DEV_LOOP_MIN_COUNT=0.