--containall causes crash error when mounting temporary file system in singularity

[cgchoza]

The below error appeared in stimela when installing the fast imaging pipeline on the Allen Telescope Array compute boxes. Singularity containers build successfully for each recipe step. The error appears when this line runs:

/usr/local/bin/singularity -d exec --containall --pwd /home/cchoza/fast_im_ata --bind /home/cchoza/fast_im_ata:/home/cchoza/fast_im_ata:rw --bind /mnt/primary/scratch/cchoza/fast_im_ms:/mnt/primary/scratch/cchoza/fast_im_ms:rw /home/cchoza/.singularity/quay.io-stimela2-casa:cc0.1.2.simg casa --log2term --nologger --nologfile -c

to run the container with casa and bind several paths into the container to give it access to necessary files and directories.

The error produced seems to deal with mounting tmpfs file systems:

DEBUG [U=1540,P=1669572] mountGeneric() Mounting tmpfs to /usr/local/var/singularity/mnt/session FATAL [U=1540,P=1669572] Master() container creation failed: mount tmpfs->/usr/local/var/singularity/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /usr/local/var/singularity/mnt/session: read unix @->@: read: connection reset by peer VERBOSE [U=0,P=1] wait_child() rpc server interrupted by signal number 11

FULL ERROR OUTPUT:

VERBOSE: Setting HOME=/home/cchoza\
VERBOSE: Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\
VERBOSE: Set messagelevel to: 4\
VERBOSE: Starter initialization\
VERBOSE: Check if we are running as setuid\
VERBOSE: Drop root privileges\
VERBOSE: Drop root privileges permanently\
VERBOSE: Spawn stage 1\
VERBOSE: Execute stage 1\
VERBOSE: stage 1 exited with status 0\
VERBOSE: Get root privileges\
VERBOSE: Change filesystem uid to 1540\
VERBOSE: Create pid namespace\
VERBOSE: Spawn master process\
VERBOSE: Create ipc namespace\
VERBOSE: Create mount namespace\
VERBOSE: Entering in mount namespace\
VERBOSE: Create mount namespace\
VERBOSE: Spawn RPC server\
VERBOSE: Execute master process\
VERBOSE: Default mount: /proc:/proc\
VERBOSE: Default mount: /sys:/sys\
VERBOSE: Default mount: /tmp:/tmp\
VERBOSE: Default mount: /var/tmp:/var/tmp\
VERBOSE: Default mount: /etc/resolv.conf:/etc/resolv.conf\
VERBOSE: rpc server interrupted by signal number 11\
FATAL:   container creation failed: mount tmpfs->/usr/local/var/singularity/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /usr/local/var/singularity/mnt/session: read unix @->@: read: connection reset by peer\
(fastim) cchoza@comp-node4:~/fast_im_ata$ /usr/local/bin/singularity -d exec --containall --pwd /home/cchoza/fast_im_ata --bind /home/cchoza/fast_im_ata:/home/cchoza/fast_im_ata:rw --bind /mnt/primary/scratch/cchoza/fast_im_ms:/mnt/primary/scratch/cchoza/fast_im_ms:rw /home/cchoza/.singularity/quay.io-stimela2-casa:cc0.1.2.simg casa --log2term --nologger --nologfile -c\
DEBUG   [U=1540,P=1669572] persistentPreRun()            Singularity version: 3.8.1\
DEBUG   [U=1540,P=1669572] persistentPreRun()            Parsing configuration file /usr/local/etc/singularity/singularity.conf\
DEBUG   [U=1540,P=1669572] handleConfDir()               /home/cchoza/.singularity already exists. Not creating.\
DEBUG   [U=1540,P=1669572] execStarter()                 Saving umask 0022 for propagation into container\
DEBUG   [U=1540,P=1669572] execStarter()                 Checking for encrypted system partition\
DEBUG   [U=1540,P=1669572] Init()                        Image format detection\
DEBUG   [U=1540,P=1669572] Init()                        Check for sandbox image format\
DEBUG   [U=1540,P=1669572] Init()                        sandbox format initializer returned: not a directory image\
DEBUG   [U=1540,P=1669572] Init()                        Check for sif image format\
DEBUG   [U=1540,P=1669572] Init()                        sif image format detected\
DEBUG   [U=1540,P=1669572] SetContainerEnv()             Forwarding TERM environment variable\
VERBOSE [U=1540,P=1669572] SetContainerEnv()             Setting HOME=/home/cchoza\
VERBOSE [U=1540,P=1669572] SetContainerEnv()             Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\
DEBUG   [U=1540,P=1669572] init()                        Use starter binary /usr/local/libexec/singularity/bin/starter-suid\
DEBUG   [U=1540,P=1669572] Exec()                        Setting GOGC=off for starter\
VERBOSE [U=0,P=1669572]    print()                       Set messagelevel to: 5\
VERBOSE [U=0,P=1669572]    init()                        Starter initialization\
DEBUG   [U=0,P=1669572]    load_overlay_module()         Trying to load overlay kernel module\
DEBUG   [U=0,P=1669572]    load_overlay_module()         Overlay seems supported by the kernel\
VERBOSE [U=0,P=1669572]    is_suid()                     Check if we are running as setuid\
VERBOSE [U=0,P=1669572]    priv_drop()                   Drop root privileges\
DEBUG   [U=1540,P=1669572]  read_engine_config()          Read engine configuration\
DEBUG   [U=1540,P=1669572]  cleanenv()                    Clearing env var LOAD_OVERLAY_MODULE=1\
DEBUG   [U=1540,P=1669572]  cleanenv()                    Keeping env var SINGULARITY_MESSAGELEVEL=5\
DEBUG   [U=1540,P=1669572]  cleanenv()                    Clearing env var ENGINE_CONFIG1=\{"engineName":"singularity","containerID":"","engineConfig":\{"jsonConfig":\{"bindpath":[\{"source":"/home/cchoza/fast_im_ata","destination":"/home/cchoza/fast_im_ata","options":\{"rw":\{\}\}\},\{"source":"/mnt/primary/scratch/cchoza/fast_im_ms","destination":"/mnt/primary/scratch/cchoza/fast_im_ms","options":\{"rw":\{\}\}\}],"unixSocketPair":[0,0],"image":"/home/cchoza/.singularity/quay.io-stimela2-casa:cc0.1.2.simg","homedir":"/home/cchoza","homeDest":"/home/cchoza","netw(TRUNCATED...)DEBUG   [U=1540,P=1669572]  cleanenv()                    Clearing env var ENGINE_CONFIG_CHUNKS=1\
DEBUG   [U=1540,P=1669572]  cleanenv()                    Keeping env var GOGC=off\
DEBUG   [U=1540,P=1669572]  init()                        Wait completion of stage1\
VERBOSE [U=1540,P=1669583]  priv_drop()                   Drop root privileges permanently\
DEBUG   [U=1540,P=1669583]  set_parent_death_signal()     Set parent death signal to 9\
VERBOSE [U=1540,P=1669583]  init()                        Spawn stage 1\
DEBUG   [U=1540,P=1669583] startup()                     singularity runtime engine selected\
VERBOSE [U=1540,P=1669583] startup()                     Execute stage 1\
DEBUG   [U=1540,P=1669583] StageOne()                    Entering stage 1\
DEBUG   [U=1540,P=1669583] prepareAutofs()               Found "/proc/sys/fs/binfmt_misc" as autofs mount point\
DEBUG   [U=1540,P=1669583] prepareAutofs()               Found "/home" as autofs mount point\
DEBUG   [U=1540,P=1669583] prepareAutofs()               Found "/mnt/datac-netStorage-1G" as autofs mount point\
DEBUG   [U=1540,P=1669583] prepareAutofs()               Found "/mnt/datad-netStorage-1G" as autofs mount point\
DEBUG   [U=1540,P=1669583] prepareAutofs()               Found "/mnt/primary" as autofs mount point\
DEBUG   [U=1540,P=1669583] prepareAutofs()               Found "/opt/mnt" as autofs mount point\
DEBUG   [U=1540,P=1669583] keepAutofsMount()             Open file descriptor for /home/cchoza/fast_im_ata\
DEBUG   [U=1540,P=1669583] keepAutofsMount()             Open file descriptor for /mnt/primary/scratch/cchoza/fast_im_ms\
DEBUG   [U=1540,P=1669583] prepareAutofs()               Could not keep file descriptor for workdir : no mount point\
DEBUG   [U=1540,P=1669583] Init()                        Image format detection\
DEBUG   [U=1540,P=1669583] Init()                        Check for sandbox image format\
DEBUG   [U=1540,P=1669583] Init()                        sandbox format initializer returned: not a directory image\
DEBUG   [U=1540,P=1669583] Init()                        Check for sif image format\
DEBUG   [U=1540,P=1669583] Init()                        sif image format detected\
DEBUG   [U=1540,P=1669583] setSessionLayer()             Overlay seems supported and allowed by kernel\
DEBUG   [U=1540,P=1669583] setSessionLayer()             Attempting to use overlayfs (enable overlay = yes)\
VERBOSE [U=1540,P=1669572]  wait_child()                  stage 1 exited with status 0\
DEBUG   [U=1540,P=1669572]  cleanup_fd()                  Setting FD_CLOEXEC on starter fd 3\
DEBUG   [U=1540,P=1669572]  cleanup_fd()                  Close file descriptor 4\
DEBUG   [U=1540,P=1669572]  cleanup_fd()                  Close file descriptor 5\
DEBUG   [U=1540,P=1669572]  cleanup_fd()                  Close file descriptor 6\
DEBUG   [U=1540,P=1669572]  cleanup_fd()                  Setting FD_CLOEXEC on starter fd 7\
DEBUG   [U=1540,P=1669572]  cleanup_fd()                  Setting FD_CLOEXEC on starter fd 8\
DEBUG   [U=1540,P=1669572]  init()                        Set child signal mask\
DEBUG   [U=1540,P=1669572]  init()                        Create socketpair for master communication channel\
DEBUG   [U=1540,P=1669572]  init()                        Create RPC socketpair for communication between stage 2 and RPC server\
VERBOSE [U=1540,P=1669572]  priv_escalate()               Get root privileges\
VERBOSE [U=0,P=1669572]    priv_escalate()               Change filesystem uid to 1540\
VERBOSE [U=0,P=1669572]    pid_namespace_init()          Create pid namespace\
VERBOSE [U=0,P=1669572]    init()                        Spawn master process\
DEBUG   [U=0,P=1]          set_parent_death_signal()     Set parent death signal to 9\
VERBOSE [U=0,P=1]          create_namespace()            Create ipc namespace\
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace\
VERBOSE [U=0,P=1669572]    enter_namespace()             Entering in mount namespace\
DEBUG   [U=0,P=1669572]    enter_namespace()             Opening namespace file ns/mnt\
VERBOSE [U=0,P=1]          create_namespace()            Create mount namespace\
DEBUG   [U=0,P=1669572]    set_master_privileges()       Set master privileges\
DEBUG   [U=0,P=1669572]    apply_privileges()            Effective capabilities:   0x00000000000000c0\
DEBUG   [U=0,P=1669572]    apply_privileges()            Permitted capabilities:   0x0000003fffffffff\
DEBUG   [U=0,P=1669572]    apply_privileges()            Bounding capabilities:    0x0000003fffffffff\
DEBUG   [U=0,P=1669572]    apply_privileges()            Inheritable capabilities: 0x0000003fffffffff\
DEBUG   [U=0,P=1669572]    apply_privileges()            Ambient capabilities:     0x0000000000000000\
DEBUG   [U=0,P=1669572]    apply_privileges()            Set user ID to 1540\
DEBUG   [U=0,P=2]          set_rpc_privileges()          Set RPC privileges\
DEBUG   [U=0,P=2]          apply_privileges()            Effective capabilities:   0x0000000000200000\
DEBUG   [U=0,P=2]          apply_privileges()            Permitted capabilities:   0x0000003fffffffff\
DEBUG   [U=0,P=2]          apply_privileges()            Bounding capabilities:    0x0000000008204000\
DEBUG   [U=0,P=2]          apply_privileges()            Inheritable capabilities: 0x0000000000000000\
DEBUG   [U=0,P=2]          apply_privileges()            Ambient capabilities:     0x0000000000000000\
DEBUG   [U=0,P=2]          apply_privileges()            Set user ID to 1540\
DEBUG   [U=1540,P=2]       set_parent_death_signal()     Set parent death signal to 9\
VERBOSE [U=1540,P=2]       init()                        Spawn RPC server\
DEBUG   [U=1540,P=1669572] startup()                     singularity runtime engine selected\
VERBOSE [U=1540,P=1669572] startup()                     Execute master process\
DEBUG   [U=1540,P=1669572] setupSessionLayout()          Using Layer system: overlay\
DEBUG   [U=1540,P=1669572] setupOverlayLayout()          Creating overlay SESSIONDIR layout\
DEBUG   [U=1540,P=1669572] addRootfsMount()              Mount rootfs in read-only mode\
DEBUG   [U=1540,P=1669572] addRootfsMount()              Image type is 4096\
DEBUG   [U=1540,P=1669572] addRootfsMount()              Mounting block [squashfs] image: /home/cchoza/.singularity/quay.io-stimela2-casa:cc0.1.2.simg\
DEBUG   [U=1540,P=1669572] addKernelMount()              Checking configuration file for 'mount proc'\
DEBUG   [U=1540,P=1669572] addKernelMount()              Adding proc to mount list\
VERBOSE [U=1540,P=1669572] addKernelMount()              Default mount: /proc:/proc\
DEBUG   [U=1540,P=1669572] addKernelMount()              Checking configuration file for 'mount sys'\
DEBUG   [U=1540,P=1669572] addKernelMount()              Adding sysfs to mount list\
VERBOSE [U=1540,P=1669572] addKernelMount()              Default mount: /sys:/sys\
DEBUG   [U=1540,P=1669572] addDevMount()                 Checking configuration file for 'mount dev'\
DEBUG   [U=1540,P=1669572] addDevMount()                 Creating temporary staged /dev\
DEBUG   [U=1540,P=1669572] addDevMount()                 Creating temporary staged /dev/shm\
DEBUG   [U=1540,P=1669572] addDevMount()                 Creating temporary staged /dev/mqueue\
DEBUG   [U=1540,P=1669572] addDevMount()                 Creating temporary staged /dev/pts\
DEBUG   [U=1540,P=1669572] addDevMount()                 Mounting devpts for staged /dev/pts\
DEBUG   [U=1540,P=1669572] addDevMount()                 Fd 0 is tty /dev/pts/5, binding to /dev/console\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Mounting device /dev/pts/5 at /usr/local/var/singularity/mnt/session/dev/console\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Mounting device /dev/tty at /usr/local/var/singularity/mnt/session/dev/tty\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Mounting device /dev/null at /usr/local/var/singularity/mnt/session/dev/null\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Mounting device /dev/zero at /usr/local/var/singularity/mnt/session/dev/zero\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Mounting device /dev/random at /usr/local/var/singularity/mnt/session/dev/random\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Mounting device /dev/urandom at /usr/local/var/singularity/mnt/session/dev/urandom\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Adding symlink device /dev/fd to /proc/self/fd at /usr/local/var/singularity/mnt/session/dev/fd\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Adding symlink device /dev/stdin to /proc/self/fd/0 at /usr/local/var/singularity/mnt/session/dev/stdin\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Adding symlink device /dev/stdout to /proc/self/fd/1 at /usr/local/var/singularity/mnt/session/dev/stdout\
DEBUG   [U=1540,P=1669572] addSessionDevAt()             Adding symlink device /dev/stderr to /proc/self/fd/2 at /usr/local/var/singularity/mnt/session/dev/stderr\
DEBUG   [U=1540,P=1669572] addHostMount()                Not mounting host file systems per configuration\
DEBUG   [U=1540,P=1669572] addBindsMount()               Binding /etc/hosts and /etc/localtime only with contain\
DEBUG   [U=1540,P=1669572] addHomeStagingDir()           Using session directory for home directory\
DEBUG   [U=1540,P=1669572] addHomeMount()                Adding home directory mount [/usr/local/var/singularity/mnt/session/home/cchoza:/home/cchoza] to list using layer: overlay\
DEBUG   [U=1540,P=1669572] addUserbindsMount()           Adding /home/cchoza/fast_im_ata to mount list\
DEBUG   [U=1540,P=1669572] addUserbindsMount()           Adding /mnt/primary/scratch/cchoza/fast_im_ms to mount list\
DEBUG   [U=1540,P=1669572] addTmpMount()                 Checking for 'mount tmp' in configuration file\
VERBOSE [U=1540,P=1669572] addTmpMount()                 Default mount: /tmp:/tmp\
VERBOSE [U=1540,P=1669572] addTmpMount()                 Default mount: /var/tmp:/var/tmp\
DEBUG   [U=1540,P=1669572] addScratchMount()             Not mounting scratch directory: Not requested\
DEBUG   [U=1540,P=1669572] addLibsMount()                Checking for 'user bind control' in configuration file\
DEBUG   [U=1540,P=1669572] addFilesMount()               Checking for 'user bind control' in configuration file\
DEBUG   [U=1540,P=1669572] addResolvConfMount()          Adding /etc/resolv.conf to mount list\
VERBOSE [U=1540,P=1669572] addResolvConfMount()          Default mount: /etc/resolv.conf:/etc/resolv.conf\
DEBUG   [U=1540,P=1669572] addHostnameMount()            Skipping hostname mount, not virtualizing UTS namespace on user request\
DEBUG   [U=1540,P=1669572] create()                      Mount all\
DEBUG   [U=1540,P=1669572] mountGeneric()                Mounting tmpfs to /usr/local/var/singularity/mnt/session\
FATAL   [U=1540,P=1669572] Master()                      container creation failed: mount tmpfs->/usr/local/var/singularity/mnt/session error: while mounting tmpfs: can't mount tmpfs filesystem to /usr/local/var/singularity/mnt/session: read unix @->@: read: connection reset by peer\
VERBOSE [U=0,P=1]          wait_child()                  rpc server interrupted by signal number 11}

ADDITIONAL INFORMATION:

1. Home directories on this compute box are NFS mounted, as is the scratch directory. Moving all folders, installs, and relevant files to a local disk, including the singularity install, has no effect on the error.
2. Changing the configured temporary file system to ramfs instead of tmpfs also has no effect on the error.
3. Whether or not the SESSIONDIR used by singularity to construct the temporary file system was owned by root or not, and the permissions on the SESSIONDIR folder, had no effect.
4. Changing the --containall flag to --contain eliminates the error and the container successfully binds paths and runs.

[o-smirnov]

I think this argues for (a) us adding a backend option to specify containment levels and (b) making --contain the sensible default. Thanks for chasing this up, @cgchoza!