Closed matthew-valentine closed 3 years ago
dtrudg
commented
3 years ago Hello, please check if you can replicate this behavior in the latest Singularity 3.7.0. If you have the same issue, please provide the entire debug output, not just the final lines.
Thanks.
matthew-valentine
commented
3 years ago I just tried it with Singularity 3.7.0. and had the same issue occur. Here is a copy of the debug output:
DEBUG [U=31325,P=43516] persistentPreRun() Singularity version: 3.7.0
DEBUG [U=31325,P=43516] persistentPreRun() Parsing configuration file /opt/local/singularity/etc/singularity/singularity.conf
DEBUG [U=31325,P=43516] handleConfDir() /home/matthew/.singularity already exists. Not creating.
DEBUG [U=31325,P=43516] execStarter() Saving umask 0022 for propagation into container
VERBOSE [U=31325,P=43516] execStarter() Unprivileged installation: using user namespace
DEBUG [U=31325,P=43516] execStarter() Checking for encrypted system partition
DEBUG [U=31325,P=43516] Init() Image format detection
DEBUG [U=31325,P=43516] Init() Check for sandbox image format
DEBUG [U=31325,P=43516] Init() sandbox format initializer returned: not a directory image
DEBUG [U=31325,P=43516] Init() Check for sif image format
DEBUG [U=31325,P=43516] Init() sif image format detected
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding SGE_CELL environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding PERL_MB_OPT environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding PWD environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding SGE_ROOT environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding PERL_MM_OPT environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding PERL_LOCAL_LIB_ROOT environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding MATLABPATH environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding TERM environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding SHELL environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding PERL5LIB environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding OCTAVE_PATH environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding SHLVL environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding MANPATH environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding LOGNAME environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding SGE_ARCH environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding SGE_CLUSTER_NAME environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding _ environment variable
DEBUG [U=31325,P=43516] SetContainerEnv() Forwarding USER_PATH environment variable
VERBOSE [U=31325,P=43516] SetContainerEnv() Setting HOME=/home/matthew
VERBOSE [U=31325,P=43516] SetContainerEnv() Setting PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE [U=31325,P=43516] execStarter() User namespace requested, convert image test.img to sandbox
INFO [U=31325,P=43516] execStarter() Converting SIF file to temporary sandbox...
DEBUG [U=31325,P=43516] Init() Image format detection
DEBUG [U=31325,P=43516] Init() Check for sandbox image format
DEBUG [U=31325,P=43516] Init() sandbox format initializer returned: not a directory image
DEBUG [U=31325,P=43516] Init() Check for sif image format
DEBUG [U=31325,P=43516] Init() sif image format detected
DEBUG [U=31325,P=43516] extract() Excluding /dev directory during root filesystem extraction (non root user)
DEBUG [U=31325,P=43516] extract() Trying unsquashfs options: [-user-xattrs -r]
DEBUG [U=31325,P=43516] unsquashfsSandboxCmd() Calling wrapped unsquashfs: singularity [-q exec --no-home --no-nv --no-rocm -C --no-init --writable -B /tmp/rootfs-594659755:/image -B /usr/bin/unsquashfs:/usr/bin/unsquashfs:ro -B /lib/x86_64-linux-gnu/libpthread.so.0:/lib/x86_64-linux-gnu/libpthread.so.0:ro -B /lib/x86_64-linux-gnu/libm.so.6:/lib/x86_64-linux-gnu/libm.so.6:ro -B /lib/x86_64-linux-gnu/libz.so.1:/lib/x86_64-linux-gnu/libz.so.1:ro -B /lib/x86_64-linux-gnu/liblzma.so.5:/lib/x86_64-linux-gnu/liblzma.so.5:ro -B /lib/x86_64-linux-gnu/liblzo2.so.2:/lib/x86_64-linux-gnu/liblzo2.so.2:ro -B /usr/lib/x86_64-linux-gnu/liblz4.so.1:/usr/lib/x86_64-linux-gnu/liblz4.so.1:ro -B /lib/x86_64-linux-gnu/libc.so.6:/lib/x86_64-linux-gnu/libc.so.6:ro -B /lib64/ld-linux-x86-64.so.2:/lib64/ld-linux-x86-64.so.2:ro -B /lib/x86_64-linux-gnu/libdl.so.2:/lib/x86_64-linux-gnu/libdl.so.2:ro /tmp/rootfs-594659755/tmp-rootfs-728826736 /usr/bin/unsquashfs -user-xattrs -r -d /image/root /image/archive-124125454 ^(.{0}[^d]|.{1}[^e]|.{2}[^v]|.{3}[^\x2f]).*$]
DEBUG [U=31325,P=43516] init() Use starter binary /opt/local/singularity/libexec/singularity/bin/starter
VERBOSE [U=31325,P=43516] print() Set messagelevel to: 5
VERBOSE [U=31325,P=43516] init() Starter initialization
VERBOSE [U=31325,P=43516] is_suid() Check if we are running as setuid
DEBUG [U=31325,P=43516] read_engine_config() Read engine configuration
DEBUG [U=31325,P=43516] init() Wait completion of stage1
DEBUG [U=31325,P=43653] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=31325,P=43653] init() Spawn stage 1
DEBUG [U=31325,P=43653] startup() singularity runtime engine selected
VERBOSE [U=31325,P=43653] startup() Execute stage 1
DEBUG [U=31325,P=43653] StageOne() Entering stage 1
DEBUG [U=31325,P=43653] prepareAutofs() No autofs mount point found
DEBUG [U=31325,P=43653] Init() Image format detection
DEBUG [U=31325,P=43653] Init() Check for sandbox image format
DEBUG [U=31325,P=43653] Init() sandbox image format detected
DEBUG [U=31325,P=43653] setSessionLayer() Using underlay layer: user namespace requested
VERBOSE [U=31325,P=43516] wait_child() stage 1 exited with status 0
DEBUG [U=31325,P=43516] init() Applying stage 1 working directory
DEBUG [U=31325,P=43516] cleanup_fd() Close file descriptor 4
DEBUG [U=31325,P=43516] cleanup_fd() Close file descriptor 5
DEBUG [U=31325,P=43516] cleanup_fd() Close file descriptor 6
DEBUG [U=31325,P=43516] init() Set child signal mask
DEBUG [U=31325,P=43516] init() Create socketpair for master communication channel
DEBUG [U=31325,P=43516] init() Create RPC socketpair for communication between stage 2 and RPC server
VERBOSE [U=31325,P=43516] user_namespace_init() Create user namespace
VERBOSE [U=31325,P=43516] create_namespace() Create user namespace
VERBOSE [U=65534,P=43516] init() Spawn master process
DEBUG [U=65534,P=43516] setup_userns_mappings() Write deny to setgroups file
DEBUG [U=65534,P=43516] setup_userns_mappings() Write to GID map
DEBUG [U=65534,P=43516] setup_userns_mappings() Write to UID map
DEBUG [U=31325,P=43659] set_parent_death_signal() Set parent death signal to 9
VERBOSE [U=31325,P=43659] create_namespace() Create mount namespace
VERBOSE [U=31325,P=43516] enter_namespace() Entering in mount namespace
DEBUG [U=31325,P=43516] enter_namespace() Opening namespace file ns/mnt
VERBOSE [U=31325,P=43659] create_namespace() Create mount namespace
VERBOSE [U=31325,P=43660] init() Spawn RPC server
DEBUG [U=31325,P=43660] startup() singularity runtime engine selected
VERBOSE [U=31325,P=43660] startup() Serve RPC requests
DEBUG [U=31325,P=43516] startup() singularity runtime engine selected
VERBOSE [U=31325,P=43516] startup() Execute master process
DEBUG [U=31325,P=43516] setupSessionLayout() Using Layer system: underlay
DEBUG [U=31325,P=43516] setupUnderlayLayout() Creating underlay SESSIONDIR layout
DEBUG [U=31325,P=43516] addRootfsMount() Mount rootfs in read-only mode
DEBUG [U=31325,P=43516] addRootfsMount() Image type is 4098
DEBUG [U=31325,P=43516] addRootfsMount() Mounting directory rootfs: /tmp/rootfs-594659755/root
DEBUG [U=31325,P=43516] addKernelMount() Checking configuration file for 'mount proc'
DEBUG [U=31325,P=43516] addKernelMount() Adding proc to mount list
VERBOSE [U=31325,P=43516] addKernelMount() Default mount: /proc:/proc
DEBUG [U=31325,P=43516] addKernelMount() Checking configuration file for 'mount sys'
DEBUG [U=31325,P=43516] addKernelMount() Adding sysfs to mount list
VERBOSE [U=31325,P=43516] addKernelMount() Default mount: /sys:/sys
DEBUG [U=31325,P=43516] addDevMount() Checking configuration file for 'mount dev'
DEBUG [U=31325,P=43516] addDevMount() Adding dev to mount list
VERBOSE [U=31325,P=43516] addDevMount() Default mount: /dev:/dev
DEBUG [U=31325,P=43516] addHostMount() Not mounting host file systems per configuration
VERBOSE [U=31325,P=43516] addBindsMount() Found 'bind path' = /etc/localtime, /etc/localtime
VERBOSE [U=31325,P=43516] addBindsMount() Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG [U=31325,P=43516] addHomeStagingDir() Staging home directory (/home/matthew) at /usr/local/singularity/var/singularity/mnt/session/home/matthew
DEBUG [U=31325,P=43516] addHomeMount() Adding home directory mount [/usr/local/singularity/var/singularity/mnt/session/home/matthew:/home/matthew] to list using layer: underlay
DEBUG [U=31325,P=43516] addTmpMount() Checking for 'mount tmp' in configuration file
VERBOSE [U=31325,P=43516] addTmpMount() Default mount: /tmp:/tmp
VERBOSE [U=31325,P=43516] addTmpMount() Default mount: /var/tmp:/var/tmp
DEBUG [U=31325,P=43516] addScratchMount() Not mounting scratch directory: Not requested
DEBUG [U=31325,P=43516] addLibsMount() Checking for 'user bind control' in configuration file
DEBUG [U=31325,P=43516] addFilesMount() Checking for 'user bind control' in configuration file
DEBUG [U=31325,P=43516] addResolvConfMount() Adding /etc/resolv.conf to mount list
VERBOSE [U=31325,P=43516] addResolvConfMount() Default mount: /etc/resolv.conf:/etc/resolv.conf
DEBUG [U=31325,P=43516] addHostnameMount() Skipping hostname mount, not virtualizing UTS namespace on user request
DEBUG [U=31325,P=43516] create() Mount all
DEBUG [U=31325,P=43516] mountGeneric() Mounting tmpfs to /usr/local/singularity/var/singularity/mnt/session
DEBUG [U=31325,P=43516] mountGeneric() Mounting /tmp/rootfs-594659755/root to /usr/local/singularity/var/singularity/mnt/session/rootfs
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/rootfs
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/rootfs
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/.singularity.d to /usr/local/singularity/var/singularity/mnt/session/underlay/.singularity.d
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/bin to /usr/local/singularity/var/singularity/mnt/session/underlay/bin
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/boot to /usr/local/singularity/var/singularity/mnt/session/underlay/boot
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/dev to /usr/local/singularity/var/singularity/mnt/session/underlay/dev
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/etc to /usr/local/singularity/var/singularity/mnt/session/underlay/etc
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/graphmap to /usr/local/singularity/var/singularity/mnt/session/underlay/graphmap
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/htslib to /usr/local/singularity/var/singularity/mnt/session/underlay/htslib
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/lib to /usr/local/singularity/var/singularity/mnt/session/underlay/lib
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/lib64 to /usr/local/singularity/var/singularity/mnt/session/underlay/lib64
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/media to /usr/local/singularity/var/singularity/mnt/session/underlay/media
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/mnt to /usr/local/singularity/var/singularity/mnt/session/underlay/mnt
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/nanopolish to /usr/local/singularity/var/singularity/mnt/session/underlay/nanopolish
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/ont-guppy to /usr/local/singularity/var/singularity/mnt/session/underlay/ont-guppy
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/opt to /usr/local/singularity/var/singularity/mnt/session/underlay/opt
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/proc to /usr/local/singularity/var/singularity/mnt/session/underlay/proc
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/root to /usr/local/singularity/var/singularity/mnt/session/underlay/root
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/run to /usr/local/singularity/var/singularity/mnt/session/underlay/run
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/samtools to /usr/local/singularity/var/singularity/mnt/session/underlay/samtools
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/sbin to /usr/local/singularity/var/singularity/mnt/session/underlay/sbin
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/srv to /usr/local/singularity/var/singularity/mnt/session/underlay/srv
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/sys to /usr/local/singularity/var/singularity/mnt/session/underlay/sys
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/tmp to /usr/local/singularity/var/singularity/mnt/session/underlay/tmp
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/usr to /usr/local/singularity/var/singularity/mnt/session/underlay/usr
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/rootfs/var to /usr/local/singularity/var/singularity/mnt/session/underlay/var
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/underlay to /usr/local/singularity/var/singularity/mnt/session/final
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/final
DEBUG [U=31325,P=43516] setPropagationMount() Set RPC mount propagation flag to SLAVE
VERBOSE [U=31325,P=43516] Passwd() Checking for template passwd file: /usr/local/singularity/var/singularity/mnt/session/rootfs/etc/passwd
VERBOSE [U=31325,P=43516] Passwd() Creating passwd content
VERBOSE [U=31325,P=43516] Passwd() Creating template passwd file and appending user data: /usr/local/singularity/var/singularity/mnt/session/rootfs/etc/passwd
DEBUG [U=31325,P=43516] addIdentityMount() Adding /etc/passwd to mount list
VERBOSE [U=31325,P=43516] addIdentityMount() Default mount: /etc/passwd:/etc/passwd
VERBOSE [U=31325,P=43516] Group() Checking for template group file: /usr/local/singularity/var/singularity/mnt/session/rootfs/etc/group
VERBOSE [U=31325,P=43516] Group() Creating group content
DEBUG [U=31325,P=43516] addIdentityMount() Adding /etc/group to mount list
VERBOSE [U=31325,P=43516] addIdentityMount() Default mount: /etc/group:/etc/group
DEBUG [U=31325,P=43516] mountGeneric() Mounting /dev to /usr/local/singularity/var/singularity/mnt/session/final/dev
DEBUG [U=31325,P=43516] mountGeneric() Mounting /etc/localtime to /usr/local/singularity/var/singularity/mnt/session/final/usr/share/zoneinfo/UCT
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/final/usr/share/zoneinfo/UCT
DEBUG [U=31325,P=43516] mountGeneric() Mounting /etc/hosts to /usr/local/singularity/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/final/etc/hosts
DEBUG [U=31325,P=43516] mountGeneric() Mounting /proc to /usr/local/singularity/var/singularity/mnt/session/final/proc
VERBOSE [U=31325,P=43516] getBindFlags() Could not mount /sys as read-write: mounted read-only
DEBUG [U=31325,P=43516] mountGeneric() Mounting /sys to /usr/local/singularity/var/singularity/mnt/session/final/sys
DEBUG [U=31325,P=43516] mountGeneric() Mounting /home/matthew to /usr/local/singularity/var/singularity/mnt/session/home/matthew
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/home/matthew
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/home/matthew to /usr/local/singularity/var/singularity/mnt/session/final/home/matthew
DEBUG [U=31325,P=43516] mountGeneric() Mounting /tmp to /usr/local/singularity/var/singularity/mnt/session/final/tmp
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/final/tmp
DEBUG [U=31325,P=43516] mountGeneric() Mounting /var/tmp to /usr/local/singularity/var/singularity/mnt/session/final/var/tmp
DEBUG [U=31325,P=43516] mountGeneric() Remounting /usr/local/singularity/var/singularity/mnt/session/final/var/tmp
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/etc/resolv.conf to /usr/local/singularity/var/singularity/mnt/session/final/etc/resolv.conf
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/etc/passwd to /usr/local/singularity/var/singularity/mnt/session/final/etc/passwd
DEBUG [U=31325,P=43516] mountGeneric() Mounting /usr/local/singularity/var/singularity/mnt/session/etc/group to /usr/local/singularity/var/singularity/mnt/session/final/etc/group
DEBUG [U=31325,P=43516] addCwdMount() Using /home/matthew as current working directory
VERBOSE [U=31325,P=43516] addCwdMount() /home/matthew found within container
DEBUG [U=31325,P=43516] create() Chroot into /usr/local/singularity/var/singularity/mnt/session/final
DEBUG [U=31325,P=43660] Chroot() Hold reference to host / directory
DEBUG [U=31325,P=43660] Chroot() Called pivot_root on /usr/local/singularity/var/singularity/mnt/session/final
DEBUG [U=31325,P=43660] Chroot() Change current directory to host / directory
DEBUG [U=31325,P=43660] Chroot() Apply slave mount propagation for host / directory
DEBUG [U=31325,P=43660] Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=31325,P=43660] Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=31325,P=43516] create() Chdir into / to avoid errors
VERBOSE [U=31325,P=43659] wait_child() rpc server exited with status 0
DEBUG [U=31325,P=43659] init() Set container privileges
DEBUG [U=31325,P=43659] apply_privileges() Effective capabilities: 0x0000000000000000
DEBUG [U=31325,P=43659] apply_privileges() Permitted capabilities: 0x0000000000000000
DEBUG [U=31325,P=43659] apply_privileges() Bounding capabilities: 0x0000000000000000
DEBUG [U=31325,P=43659] apply_privileges() Inheritable capabilities: 0x0000000000000000
DEBUG [U=31325,P=43659] apply_privileges() Ambient capabilities: 0x0000000000000000
DEBUG [U=31325,P=43659] apply_privileges() Set user ID to 31325
DEBUG [U=31325,P=43659] set_parent_death_signal() Set parent death signal to 9
DEBUG [U=31325,P=43659] startup() singularity runtime engine selected
VERBOSE [U=31325,P=43659] startup() Execute stage 2
DEBUG [U=31325,P=43659] StageTwo() Entering stage 2
DEBUG [U=31325,P=43659] StartProcess() Setting umask in container to 0022
DEBUG [U=31325,P=43659] sylogBuiltin() Sourcing /.singularity.d/env/01-base.sh
DEBUG [U=31325,P=43659] sylogBuiltin() Sourcing /.singularity.d/env/10-docker2singularity.sh
DEBUG [U=31325,P=43659] sylogBuiltin() Sourcing /.singularity.d/env/90-environment.sh
DEBUG [U=31325,P=43659] sylogBuiltin() Sourcing /.singularity.d/env/94-appsbase.sh
DEBUG [U=31325,P=43659] sylogBuiltin() Sourcing /.singularity.d/env/95-apps.sh
DEBUG [U=31325,P=43659] sylogBuiltin() Sourcing /.singularity.d/env/99-base.sh
DEBUG [U=31325,P=43659] sylogBuiltin() Sourcing /.singularity.d/env/99-runtimevars.sh
DEBUG [U=31325,P=43659] sylogBuiltin() Running action command exec
DEBUG [U=31325,P=43516] PostStartProcess() Post start process
test
VERBOSE [U=31325,P=43516] CleanupContainer() Removing image tempDir /tmp/rootfs-594659755
INFO [U=31325,P=43516] CleanupContainer() Cleaning up image...
DEBUG [U=31325,P=43516] Master() Child exited with exit status 0I also received some information about the cluster nodes. The cluster nodes are run using lxd with privilege to run nesting container.
raw.lxc: lxc.apparmor.profile=unconfined
security.nesting: "true"
security.privileged: "true"The singularity configure option is:
./mconfig --prefix=/opt/local/singularity --without-suid --localstatedir=/usr/local/singularity/var@matthew-valentine does this happen every time with this container? On the server where you replicate are you using lxd also?
What would be useful here is running singularity under strace -f - or looking for the process ID for the stuck process and doing strace -p <pid> to see what it's stuck on.
dtrudg
commented
3 years ago Haven't been able to replicate this using singularity in user-namespace / unpriv mode. Will need to know exactly what lxd setup this is running under in detail in order to replicate.
matthew-valentine
commented
3 years ago I have asked for some more information about the server and I'll get back to you when I hear more. It happens every time when I run this container on this particular server, either when I run it directly from the command line or if running with qsub (which was where I originally discovered the error). We have another server for running code on to test it before running it via qsub and it works fine on there, so I'm trying to find out some information about that server too.
I just tried running it with strace -f. Do you need the whole output from that as it is very long, or just the last bit? I shall post a portion here, but let me know if you need more and I can provide it.
[pid 28355] umask(022) = 022
[pid 28355] newfstatat(AT_FDCWD, "/tmp", {st_mode=S_IFDIR|S_ISVTX|0777, st_size=327680, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid 28355] newfstatat(AT_FDCWD, "/tmp/rootfs-954799678", {st_mode=S_IFDIR|0700, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid 28355] newfstatat(AT_FDCWD, "/tmp/rootfs-954799678/tmp-rootfs-322107135", {st_mode=S_IFDIR|0700, st_size=4096, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid 28355] openat(AT_FDCWD, "/tmp/rootfs-954799678/tmp-rootfs-322107135", O_RDONLY|O_CLOEXEC) = 4
[pid 28355] epoll_ctl(5, EPOLL_CTL_ADD, 4, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2140335896, u64=22610848182040}}) = -1 EPERM (Operation not permitted)
[pid 28355] epoll_ctl(5, EPOLL_CTL_DEL, 4, 0xc00061f1dc) = -1 EPERM (Operation not permitted)
[pid 28355] fstat(4, {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
[pid 28355] fcntl(4, F_SETFD, 0x80000 /* FD_??? */) = 0
[pid 28355] close(4) = 0
[pid 28355] newfstatat(AT_FDCWD, ".", {st_mode=S_IFDIR|0755, st_size=29, ...}, 0) = 0
[pid 28355] getcwd("/", 4096) = 2
[pid 28355] newfstatat(AT_FDCWD, "/tmp/rootfs-954799678/tmp-rootfs-322107135", {st_mode=S_IFDIR|0700, st_size=4096, ...}, 0) = 0
[pid 28355] newfstatat(AT_FDCWD, "/opt/local/singularity/libexec/singularity/plugin", 0xc00019fca8, 0) = -1 ENOENT (No such file or directory)
[pid 28355] newfstatat(AT_FDCWD, "/opt/local/singularity/libexec/singularity/bin/starter", <unfinished ...>
[pid 28349] <... nanosleep resumed> NULL) = 0
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28349] futex(0x559671982758, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=7916322} <unfinished ...>
[pid 28352] <... epoll_pwait resumed> [], 128, 17, NULL, 0) = 0
[pid 28352] epoll_pwait(5, <unfinished ...>
[pid 28349] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid 28349] futex(0xc000111648, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28357] <... futex resumed> ) = 0
[pid 28352] <... epoll_pwait resumed> [], 128, 1, NULL, 0) = 0
[pid 28349] <... futex resumed> ) = 1
[pid 28357] futex(0xc000110bc8, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 28352] futex(0xc000180148, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28357] <... futex resumed> ) = 1
[pid 28351] <... futex resumed> ) = 0
[pid 28357] madvise(0xc000400000, 2097152, MADV_NOHUGEPAGE <unfinished ...>
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 28357] <... madvise resumed> ) = 0
[pid 28351] futex(0xc000110bc8, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid 28357] madvise(0xc0005cc000, 8192, MADV_FREE) = 0
[pid 28357] epoll_pwait(5, <unfinished ...>
[pid 28349] <... nanosleep resumed> NULL) = 0
[pid 28349] futex(0x559671982758, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=10630117} <unfinished ...>
[pid 28355] <... newfstatat resumed> {st_mode=S_IFREG|0755, st_size=26022424, ...}, 0) = 0
[pid 28355] futex(0x559671982758, FUTEX_WAKE_PRIVATE, 1) = 1
[pid 28349] <... futex resumed> ) = 0
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 28355] getrlimit(RLIMIT_STACK, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
[pid 28349] <... nanosleep resumed> NULL) = 0
[pid 28355] execve("/opt/local/singularity/libexec/singularity/bin/starter", ["Singularity runtime parent"], [/* 3 vars */] <unfinished ...>
[pid 28349] nanosleep({tv_sec=0, tv_nsec=20000}, NULL) = 0I haven't managed to get any additional information about the server yet, but I just tried running it again on the same server and it seems to not get stuck anymore. As far as I am aware there have been no changes to the server, so I don't know why it is now working, but I'm not going to complain about it! Unfortunately (for me) I thought this singularity hanging issue was the source of the woes I was having with an application using singularity, but it seems it isn't because I am still having issues now singularity is able to complete. Back to the drawing board I guess.
dtrudg
commented
3 years ago Thanks for the update @matthew-valentine - I'm glad things are working for you even if the reason isn't understood. We'll keep this open for a while in case it comes up again.
Version of Singularity:
Running singularity on a server (build below), most often run via qsub (SGE) but for the purposes of testing I have just been running directly on the server.
Expected behavior
Running the following:
Should output
And then it should finish running
Actual behavior
The code executes correctly, however, the process never completes. Running it with -d I get:
It just hangs here for as long as I let it run for
Steps to reproduce this behavior
How did you install Singularity
Singularity was already installed on the server