home mount erase /usr/ in container

[mschnepf]

Hi I’m using singularity 2.3, also tested with 2.2.3, on Ubuntu 16.04. Our home directory is on /usr/users/${user}.

I start a singulartiy container with disabled overlay file system (enable overlay = no) and enabled home mount (mount home = yes). In this container /usr/ is complete empty excluded /usr/users/${user}. So /usr/bin/ /usr/lib/ ... is no longer available inside the container. With enabled overlay file system all directories in /usr/ and the home (/usr/users/${user}) is available inside the container.

This makes it impossible with disables overlay file system to use programs which are installed in /usr/bin/ for us.

[vsoch]

When you disable overlayfs, this means that any path in the container that doesn't exist (but exists for the user on the host) will not be automatically generated. Given that, it makes sense that the container doesn't mount anything under /usr/users, that's not a standard location on most linux distributions (at least not on the few I've used, haha). There are a couple of things you could try.

1. You could try generating container that does have the folder /usr/users and then enable mount home = yes and enable overlay = no, and see if the home is allowed to be mounted.
2. If that doesn't work, you could add /usr/users as an allowed mount point and try again (without overlayfs).
3. Finally, you can try going one level up, creating a single user folder, and seeing if that works (still without overlayfs).

Just curious - why did you put home directories in /usr/users? What do you have under /home? And what is the concern about using overlayfs? My (probably somewhat poor) understanding of /usr is that it is primarily for software for users, not personal files / home directories. It would be like finding someone put their dressing table in the garage workshop, haha.

Anyway, if the above three things don't work, please run a (failed) command with singularity --debug so we can see the order of operations of testing mount points, and exact error messages.

[gmkurtzer]

Hi @mschnepf,

Oh gosh... I haven't seen anyone putting home directories in /usr/users for a very very very long time! Due to how Singularity 'tries' to be as smart as possible, with regards to home directory mounts when overlayfs is not available, there is no direct workaround for this at present.

An indirect workaround may exist if the prctl() option SET_NO_NEW_PRIVS is available on your distribution. This would enable you to virtualize your home directory. Check out the -H feature in version 2.3 and see if you can use this to 'move' your home directory. If so, you can set this to be automatic via the environment variable SINGULARITY_HOME=/source:/dest.

Also, just to make sure there isn't another bug that might be triggering this, does your system support overlayFS? You can find it in the --debug output and/or in via the ./configure build output.

Thanks!

[mschnepf]

Hi

The home directory is mounted via NFS to /usr/users/ on all our machines. The /home/ is the local home only for that machine. Thanks for your ideas. I created in the container/usr/users/ and /usr/users/mschnepf. After that, I started the container with overlay = no. In both cases the home direct was mounted but /usr/bin, /usr/lib, …. was always not existing in the container.

I can move the home directory with the-H option to another directory. With this option, all directories in /usr/ are in the container. However, this would not really help. Some machines are integrated in out batch system and has to move /home/somewhere/else to /usr/users/. In this case, /usr/ is empty expect /usr/users/mschnepf.

I looked at the output of ./configureand found checking for overlayfs... yes. So my system should support overlayFS.

[gmkurtzer]

Hi @mschnepf,

In looking through the source code of version 2.3, it appears that if overlayFS is indeed enabled, there is no staging of the home directory which would account for the apparent loss of the /usr directory. Can you send me the debug output of a normal run with OverlayFS enabled and no -H option?

Thank you!

[mschnepf]

Hi,

here the debug output with 'enable overlay = yes' where the mount work fine:

`singularity --debug shell /local/scratch/ssd/mschnepf/slc6-pilot.img Enabling debugging Ending argument loop Singularity version: 2.3-dist Exec'ing: /usr/local/libexec/singularity/cli/shell.exec Evaluating args: '/local/scratch/ssd/mschnepf/slc6-pilot.img' VERBOSE [U=0,P=114529] message_init() Set messagelevel to: 5 VERBOSE [U=0,P=114529] singularity_config_parse() Initialize configuration file: /usr/local/etc/singularity/singularity.conf DEBUG [U=0,P=114529] singularity_config_parse() Starting parse of configuration file /usr/local/etc/singularity/singularity.conf VERBOSE [U=0,P=114529] singularity_config_parse() Got config key allow setuid = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key max loop devices = '256' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key allow pid ns = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key config passwd = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key config group = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key config resolv_conf = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key mount proc = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key mount sys = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key mount dev = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key mount home = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key mount tmp = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key mount hostfs = 'no' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key bind path = '/etc/localtime' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key bind path = '/etc/hosts' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key user bind control = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key enable overlay = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key mount slave = 'yes' VERBOSE [U=0,P=114529] singularity_config_parse() Got config key sessiondir max size = '16' DEBUG [U=0,P=114529] singularity_config_parse() Finished parsing configuration file '/usr/local/etc/singularity/singularity.conf' VERBOSE [U=0,P=114529] singularity_registry_init() Initializing Singularity Registry VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(libexecdir, /usr/local/libexec) = 0 VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'COMMAND' = 'shell' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(COMMAND, shell) = 0 VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'MESSAGELEVEL' = '5' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(MESSAGELEVEL, 5) = 0 VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'VERSION' = '2.3-dist' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(version, 2.3-dist) = 0 VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(localstatedir, /usr/local/var) = 0 VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(sysconfdir, /usr/local/etc) = 0 VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'BINDIR' = '/usr/local/bin' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(bindir, /usr/local/bin) = 0 VERBOSE [U=0,P=114529] singularity_registry_set() Adding value to registry: 'IMAGE' = '/local/scratch/ssd/mschnepf/slc6-pilot.img' DEBUG [U=0,P=114529] singularity_registry_set() Returning singularity_registry_set(IMAGE, /local/scratch/ssd/mschnepf/slc6-pilot.img) = 0 DEBUG [U=0,P=114529] singularity_registry_get() Returning NULL on 'HOME' DEBUG [U=0,P=114529] singularity_registry_get() Returning NULL on 'TARGET_UID' DEBUG [U=0,P=114529] singularity_registry_get() Returning NULL on 'TARGET_GID' DEBUG [U=0,P=114529] singularity_priv_init() Initializing user info DEBUG [U=0,P=114529] singularity_priv_init() Set the calling user's username to: mschnepf DEBUG [U=0,P=114529] singularity_priv_init() Marking uinfo structure as ready DEBUG [U=0,P=114529] singularity_priv_init() Obtaining home directory VERBOSE [U=0,P=114529] singularity_priv_init() Set home (via getpwuid()) to: /usr/users/mschnepf VERBOSE [U=0,P=114529] singularity_suid_init() Running SUID program workflow VERBOSE [U=0,P=114529] singularity_suid_init() Checking program has appropriate permissions VERBOSE [U=0,P=114529] singularity_suid_init() Checking configuration file is properly owned by root VERBOSE [U=0,P=114529] singularity_suid_init() Checking if singularity.conf allows us to run as suid DEBUG [U=0,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow setuid, yes) DEBUG [U=0,P=114529] singularity_config_get_value_impl() Returning configuration value allow setuid='yes' DEBUG [U=0,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow setuid, yes) = 1 DEBUG [U=0,P=114529] singularity_registry_get() Returning NULL on 'NOSUID' VERBOSE [U=0,P=114529] singularity_priv_userns() Invoking the user namespace DEBUG [U=0,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow user ns, yes) DEBUG [U=0,P=114529] singularity_config_get_value_impl() No configuration entry found for 'allow user ns'; returning default value 'yes' DEBUG [U=0,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow user ns, yes) = 1 VERBOSE [U=0,P=114529] singularity_priv_userns() Not virtualizing USER namespace: running as SUID DEBUG [U=0,P=114529] singularity_priv_userns() Returning singularity_priv_init(void) DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_ns() Calling: _singularity_runtime_ns_pid() DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow pid ns, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value allow pid ns='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow pid ns, yes) = 1 DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'UNSHARE_PID' VERBOSE [U=12089,P=114529] singularity_runtime_ns_pid() Not virtualizing PID namespace on user request DEBUG [U=12089,P=114529] singularity_runtime_ns() Calling: _singularity_runtime_ns_ipc() DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow ipc ns, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() No configuration entry found for 'allow ipc ns'; returning default value 'yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow ipc ns, yes) = 1 DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'UNSHARE_IPC' VERBOSE [U=12089,P=114529] singularity_runtime_ns_ipc() Not virtualizing IPC namespace on user request DEBUG [U=12089,P=114529] singularity_runtime_ns() Calling: _singularity_runtime_ns_mnt() DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount slave, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount slave='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount slave, yes) = 1 DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114529] singularity_runtime_ns_mnt() Virtualizing FS namespace DEBUG [U=0,P=114529] singularity_runtime_ns_mnt() Virtualizing mount namespace DEBUG [U=0,P=114529] singularity_runtime_ns_mnt() Making mounts slave DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_sessiondir() Setting sessiondir VERBOSE [U=12089,P=114529] singularity_sessiondir() Using session directory: /usr/local/var/singularity/mnt/session DEBUG [U=12089,P=114529] singularity_sessiondir() Checking for session directory: /usr/local/var/singularity/mnt/session DEBUG [U=12089,P=114529] singularity_sessiondir() Obtaining the default sessiondir size DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value sessiondir max size='16' DEBUG [U=12089,P=114529] singularity_sessiondir() Converted sessiondir size to: 16 DEBUG [U=12089,P=114529] singularity_sessiondir() Creating the sessiondir size mount option length DEBUG [U=12089,P=114529] singularity_sessiondir() Got size length of: 9 DEBUG [U=12089,P=114529] singularity_sessiondir() Creating the sessiondir size mount option string DEBUG [U=12089,P=114529] singularity_sessiondir() Checking to make sure the string was allocated correctly DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114529] singularity_sessiondir() Mounting sessiondir tmpfs: /usr/local/var/singularity/mnt/session DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID VERBOSE [U=12089,P=114529] singularity_registry_set() Adding value to registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114529] singularity_registry_set() Returning singularity_registry_set(SESSIONDIR, /usr/local/var/singularity/mnt/session) = 0 DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CLEANUPDIR' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'NOSESSIONCLEANUP' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'NOCLEANUP' DEBUG [U=12089,P=114529] singularity_cleanupd() Not running a cleanup thread, no 'SINGULARITY_CLEANUPDIR' defined DEBUG [U=12089,P=114529] singularity_registry_get() Returning value from registry: 'IMAGE' = '/local/scratch/ssd/mschnepf/slc6-pilot.img' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'WRITABLE' DEBUG [U=12089,P=114529] singularity_config_get_value_impl() No configuration entry found for 'limit container owners'; returning default value 'NULL' DEBUG [U=12089,P=114529] singularity_config_get_value_impl() No configuration entry found for 'limit container paths'; returning default value 'NULL' DEBUG [U=12089,P=114529] singularity_image_open() Opening file descriptor to image: /local/scratch/ssd/mschnepf/slc6-pilot.img VERBOSE [U=12089,P=114529] singularity_image_check() Checking that file pointer is a Singularity image DEBUG [U=12089,P=114529] singularity_image_check() First line of image(fd=4): #!/usr/bin/env run-singularity

DEBUG [U=12089,P=114529] singularity_image_check() Checking if first line matches key VERBOSE [U=12089,P=114529] singularity_image_check() File is a valid Singularity image DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value max loop devices='256' DEBUG [U=12089,P=114529] singularity_image_bind() Entered singularity_image_bind() DEBUG [U=12089,P=114529] singularity_image_bind() Converting max_loop_devs_string to int: '256' DEBUG [U=12089,P=114529] singularity_image_bind() Converted max_loop_devs_string to int: '256' -> 256 DEBUG [U=12089,P=114529] singularity_image_bind() Checking if this image has been properly opened DEBUG [U=12089,P=114529] singularity_image_bind() Checking if image is valid file DEBUG [U=12089,P=114529] singularity_image_bind() Checking if image is already bound to a loop device DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'WRITABLE' DEBUG [U=12089,P=114529] singularity_image_bind() Setting loopdev open to: O_RDONLY DEBUG [U=12089,P=114529] singularity_image_bind() Setting LO_FLAGS_AUTOCLEAR DEBUG [U=12089,P=114529] singularity_image_bind() Calculating image offset VERBOSE [U=12089,P=114529] singularity_image_check() Checking that file pointer is a Singularity image DEBUG [U=12089,P=114529] singularity_image_check() First line of image(fd=4): #!/usr/bin/env run-singularity

DEBUG [U=12089,P=114529] singularity_image_check() Checking if first line matches key VERBOSE [U=12089,P=114529] singularity_image_check() File is a valid Singularity image VERBOSE [U=12089,P=114529] singularity_image_offset() Calculating image offset VERBOSE [U=12089,P=114529] singularity_image_offset() Found image at an offset of 31 bytes DEBUG [U=12089,P=114529] singularity_image_offset() Returning image_offset(image_fp) = 31 DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114529] singularity_image_bind() Finding next available loop device... VERBOSE [U=0,P=114529] singularity_image_bind() Found available loop device: /dev/loop0 DEBUG [U=0,P=114529] singularity_image_bind() Setting loop device flags DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID VERBOSE [U=12089,P=114529] singularity_image_bind() Using loop device: /dev/loop0 VERBOSE [U=12089,P=114529] singularity_runtime_rootfs() Set container directory to: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Checking for container directory DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114529] chk_mode() Checking exact mode (40755) on: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114529] chk_mode() Found appropriate mode on file: /usr/local/var/singularity/mnt/container VERBOSE [U=12089,P=114529] singularity_image_mount() Checking what kind of image we are mounting DEBUG [U=12089,P=114529] singularity_image_mount_squashfs_check() Image does not appear to be of type '.sqsh': /local/scratch/ssd/mschnepf/slc6-pilot.img DEBUG [U=12089,P=114529] singularity_image_mount_dir_check() Checking if image is a directory VERBOSE [U=12089,P=114529] singularity_image_mount_dir_check() Source path is not a directory: /local/scratch/ssd/mschnepf/slc6-pilot.img VERBOSE [U=12089,P=114529] singularity_image_mount() Attempting to mount as singularity image DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'WRITABLE' DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_image_mount_image_mount() Mounting /dev/loop0 in read/only to: /usr/local/var/singularity/mnt/container DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114529] action_ready() Checking if container is valid at: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114529] singularity_runtime_overlayfs() Checking if overlayfs should be used DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(enable overlay, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value enable overlay='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(enable overlay, yes) = 1 DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'DISABLE_OVERLAYFS' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'WRITABLE' DEBUG [U=12089,P=114529] singularity_runtime_overlayfs() OverlayFS enabled by host build DEBUG [U=12089,P=114529] singularity_runtime_overlayfs() Setting up overlay mount options DEBUG [U=12089,P=114529] singularity_runtime_overlayfs() Checking for existance of overlay directory: /usr/local/var/singularity/mnt/overlay DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114529] singularity_runtime_overlayfs() Mounting overlay tmpfs: /usr/local/var/singularity/mnt/overlay DEBUG [U=0,P=114529] singularity_runtime_overlayfs() Creating upper overlay directory: /usr/local/var/singularity/mnt/overlay/upper DEBUG [U=0,P=114529] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/overlay/upper DEBUG [U=0,P=114529] singularity_runtime_overlayfs() Creating overlay work directory: /usr/local/var/singularity/mnt/overlay/work DEBUG [U=0,P=114529] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/overlay/work DEBUG [U=0,P=114529] singularity_runtime_overlayfs() Creating overlay_final directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=0,P=114529] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/overlay/final VERBOSE [U=0,P=114529] singularity_runtime_overlayfs() Mounting overlay with options: lowerdir=/usr/local/var/singularity/mnt/container,upperdir=/usr/local/var/singularity/mnt/overlay/upper,workdir=/usr/local/var/singularity/mnt/overlay/work DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID VERBOSE [U=12089,P=114529] singularity_registry_set() Adding value to registry: 'OVERLAYFS_ENABLED' = '1' DEBUG [U=12089,P=114529] singularity_registry_set() Returning singularity_registry_set(OVERLAYFS_ENABLED, 1) = 0 VERBOSE [U=12089,P=114529] singularity_runtime_overlayfs() Updating the containerdir to: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Setting container_directory = '/usr/local/var/singularity/mnt/overlay/final' DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final VERBOSE [U=12089,P=114529] singularity_runtime_mounts() Running all mount components DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount hostfs, no) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount hostfs='no' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount hostfs, no) = 0 DEBUG [U=12089,P=114529] singularity_runtime_mount_hostfs() Not mounting host file systems per configuration DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114529] singularity_runtime_mount_binds() Checking configuration file for 'bind path' VERBOSE [U=12089,P=114529] singularity_runtime_mount_binds() Found 'bind path' = /etc/localtime, /etc/localtime DEBUG [U=12089,P=114529] singularity_runtime_mount_binds() Checking if bind point is already mounted: /etc/localtime DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114529] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_binds() Binding '/etc/localtime' to '/usr/local/var/singularity/mnt/overlay/final//etc/localtime' DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID VERBOSE [U=12089,P=114529] singularity_runtime_mount_binds() Found 'bind path' = /etc/hosts, /etc/hosts DEBUG [U=12089,P=114529] singularity_runtime_mount_binds() Checking if bind point is already mounted: /etc/hosts DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114529] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_binds() Binding '/etc/hosts' to '/usr/local/var/singularity/mnt/overlay/final//etc/hosts' DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_runtime_mount_kernelfs() Checking configuration file for 'mount proc' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount proc, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount proc='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount proc, yes) = 1 DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'PIDNS_ENABLED' DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_kernelfs() Bind-mounting host /proc DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_mount_kernelfs() Checking configuration file for 'mount sys' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount sys, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount sys='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount sys, yes) = 1 DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_kernelfs() Mounting /sys DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount dev='yes' DEBUG [U=12089,P=114529] singularity_runtime_mount_dev() Checking configuration file for 'mount dev' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount dev, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount dev='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount dev, yes) = 1 DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_dev() Bind mounting /dev DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount home, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount home='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount home, yes) = 1 DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Checking that home directry is configured: /usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Checking if SINGULARITY_CONTAIN is set DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Checking if home directories are being influenced by user DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'HOME' DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Checking ownership of home directory source: /usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Checking to make sure home directory destination is a full path: /usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Checking if home directory is already mounted: /usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114529] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Checking if overlay is enabled DEBUG [U=12089,P=114529] singularity_registry_get() Returning value from registry: 'OVERLAYFS_ENABLED' = '1' DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Binding home directory direct (no staging) DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Setting home directory source to: '/usr/users/mschnepf' DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Setting home directory dest to: '/usr/local/var/singularity/mnt/overlay/final' + '/usr/users/mschnepf' DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Creating home directry within container: /usr/local/var/singularity/mnt/overlay/final/usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114529] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/overlay/final/usr/users DEBUG [U=0,P=114529] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/overlay/final/usr/users/mschnepf DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_mount_home() Created home directory within the container: /usr/local/var/singularity/mnt/overlay/final/usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_home() Mounting home directory source into container: /usr/users/mschnepf -> /usr/local/var/singularity/mnt/overlay/final/usr/users/mschnepf DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] envar_set() Setting environment variable: 'HOME' = '/usr/users/mschnepf' DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_runtime_mount_userbinds() Checking for environment variable 'SINGULARITY_BINDPATH' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'BINDPATH' DEBUG [U=12089,P=114529] singularity_runtime_mount_userbinds() No user bind mounts specified. DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount tmp, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value mount tmp='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount tmp, yes) = 1 DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114529] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_tmp() Mounting directory: /tmp DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114529] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_mount_tmp() Mounting directory: /var/tmp DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_runtime_mount_scratch() Getting SINGULARITY_SCRATCHDIR from environment DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'SCRATCHDIR' DEBUG [U=12089,P=114529] singularity_runtime_mount_scratch() Not mounting scratch directory: Not requested DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_runtime_mount_cwd() Checking to see if we should mount current working directory DEBUG [U=12089,P=114529] singularity_runtime_mount_cwd() Getting current working directory DEBUG [U=12089,P=114529] singularity_runtime_mount_cwd() Checking if current directory exists in container DEBUG [U=12089,P=114529] file_devino() Called file_devino(/usr/users/mschnepf) DEBUG [U=12089,P=114529] file_devino() Returning file_devino(/usr/users/mschnepf) = 49.23068673 DEBUG [U=12089,P=114529] file_devino() Called file_devino(/usr/local/var/singularity/mnt/overlay/final/usr/users/mschnepf) DEBUG [U=12089,P=114529] file_devino() Returning file_devino(/usr/local/var/singularity/mnt/overlay/final/usr/users/mschnepf) = 49.23068673 DEBUG [U=12089,P=114529] singularity_runtime_mount_cwd() Checking if container's cwd == host's cwd VERBOSE [U=12089,P=114529] singularity_runtime_mount_cwd() Not mounting current directory: location already available within container DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final VERBOSE [U=12089,P=114529] singularity_runtime_files() Running file components DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_registry_get() Returning value from registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114529] singularity_runtime_files_passwd() Called singularity_file_passwd_create() DEBUG [U=12089,P=114529] singularity_runtime_files_passwd() Checking configuration option: 'config passwd' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(config passwd, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value config passwd='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(config passwd, yes) = 1 VERBOSE [U=12089,P=114529] singularity_runtime_files_passwd() Checking for template passwd file: /usr/local/var/singularity/mnt/overlay/final/etc/passwd VERBOSE [U=12089,P=114529] singularity_runtime_files_passwd() Creating template of /etc/passwd DEBUG [U=12089,P=114529] copy_file() Called copy_file(/usr/local/var/singularity/mnt/overlay/final/etc/passwd, /usr/local/var/singularity/mnt/session/passwd) DEBUG [U=12089,P=114529] copy_file() Opening source file: /usr/local/var/singularity/mnt/overlay/final/etc/passwd DEBUG [U=12089,P=114529] copy_file() Opening destination file: /usr/local/var/singularity/mnt/session/passwd DEBUG [U=12089,P=114529] copy_file() Calling fstat() on source file descriptor: 6 DEBUG [U=12089,P=114529] copy_file() Cloning permission string of source to dest DEBUG [U=12089,P=114529] copy_file() Copying file data... DEBUG [U=12089,P=114529] copy_file() Done copying data, closing file pointers DEBUG [U=12089,P=114529] copy_file() Returning copy_file(/usr/local/var/singularity/mnt/overlay/final/etc/passwd, /usr/local/var/singularity/mnt/session/passwd) = 0 VERBOSE [U=12089,P=114529] singularity_runtime_files_passwd() Creating template passwd file and appending user data: /usr/local/var/singularity/mnt/session/passwd DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] container_file_bind() Called file_bind(/usr/local/var/singularity/mnt/session/passwd, /etc/passwd() DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] container_file_bind() Binding file '/usr/local/var/singularity/mnt/session/passwd' to '/usr/local/var/singularity/mnt/overlay/final/etc/passwd' DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] envar_set() Setting environment variable: 'HOME' = '/usr/users/mschnepf' DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_registry_get() Returning value from registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114529] singularity_runtime_files_group() Called singularity_file_group_create() DEBUG [U=12089,P=114529] singularity_runtime_files_group() Checking configuration option: 'config group' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(config group, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value config group='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(config group, yes) = 1 VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Creating template of /etc/group for containment DEBUG [U=12089,P=114529] copy_file() Called copy_file(/usr/local/var/singularity/mnt/overlay/final/etc/group, /usr/local/var/singularity/mnt/session/group) DEBUG [U=12089,P=114529] copy_file() Opening source file: /usr/local/var/singularity/mnt/overlay/final/etc/group DEBUG [U=12089,P=114529] copy_file() Opening destination file: /usr/local/var/singularity/mnt/session/group DEBUG [U=12089,P=114529] copy_file() Calling fstat() on source file descriptor: 6 DEBUG [U=12089,P=114529] copy_file() Cloning permission string of source to dest DEBUG [U=12089,P=114529] copy_file() Copying file data... DEBUG [U=12089,P=114529] copy_file() Done copying data, closing file pointers DEBUG [U=12089,P=114529] copy_file() Returning copy_file(/usr/local/var/singularity/mnt/overlay/final/etc/group, /usr/local/var/singularity/mnt/session/group) = 0 VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Updating group file with user info DEBUG [U=12089,P=114529] singularity_runtime_files_group() Getting supplementary group info VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Found supplementary group membership in: 132 VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Adding user's supplementary group ('docker') info to template group file VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Found supplementary group membership in: 500 VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Adding user's supplementary group ('printer') info to template group file DEBUG [U=12089,P=114529] singularity_runtime_files_group() Skipping duplicate supplementary group VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Found supplementary group membership in: 1900 VERBOSE [U=12089,P=114529] singularity_runtime_files_group() Adding user's supplementary group ('ekpadmin') info to template group file DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] container_file_bind() Called file_bind(/usr/local/var/singularity/mnt/session/group, /etc/group() DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] container_file_bind() Binding file '/usr/local/var/singularity/mnt/session/group' to '/usr/local/var/singularity/mnt/overlay/final/etc/group' DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_files_resolvconf() Checking configuration option DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(config resolv_conf, yes) DEBUG [U=12089,P=114529] singularity_config_get_value_impl() Returning configuration value config resolv_conf='yes' DEBUG [U=12089,P=114529] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(config resolv_conf, yes) = 1 DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] container_file_bind() Called file_bind(/etc/resolv.conf, /etc/resolv.conf() DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] container_file_bind() Binding file '/etc/resolv.conf' to '/usr/local/var/singularity/mnt/overlay/final/etc/resolv.conf' DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_registry_get() Returning value from registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CONTAINLIBS' DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final VERBOSE [U=12089,P=114529] singularity_runtime_enter() Containing all rootfs components DEBUG [U=12089,P=114529] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114529] singularity_runtime_enter_chroot() Entering container file system root: /usr/local/var/singularity/mnt/overlay/final DEBUG [U=0,P=114529] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114529] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114529] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114529] singularity_runtime_enter_chroot() Changing dir to '/' within the new root DEBUG [U=12089,P=114529] singularity_runtime_environment() Cloning environment DEBUG [U=12089,P=114529] singularity_runtime_environment() Counted 37 environment elements DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CLEANENV' DEBUG [U=12089,P=114529] singularity_runtime_environment() Cleaning environment DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_PAPER=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_ADDRESS=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: XDG_SESSION_ID=939 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_MONETARY=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: TERM=xterm-256color DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SHELL=/bin/bash DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SSH_CLIENT=172.23.242.246 33654 22 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_NUMERIC=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SSH_TTY=/dev/pts/8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_libexecdir=/usr/local/libexec DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_libexecdir DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_COMMAND=shell DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_COMMAND DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: USER=mschnepf DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_MESSAGELEVEL=5 DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_MESSAGELEVEL DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_TELEPHONE=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SSH_AUTH_SOCK=/tmp/ssh-xBnNIKo5je/agent.114196 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_version=2.3-dist DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_version DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: MAIL=/var/mail/mschnepf DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_localstatedir=/usr/local/var DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_localstatedir DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: QT_QPA_PLATFORMTHEME=appmenu-qt5 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_IDENTIFICATION=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: PWD=/usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITYENV_HISTFILE= DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITYENV_PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LANG=en_US.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_MEASUREMENT=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SHLVL=1 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: HOME=/usr/users/mschnepf DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_sysconfdir=/usr/local/etc DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_sysconfdir DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LOGNAME=mschnepf DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_bindir=/usr/local/bin DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_bindir DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SSH_CONNECTION=172.23.242.246 33654 129.13.102.147 22 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: XDG_RUNTIME_DIR=/run/user/12089 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: DISPLAY=localhost:10.0 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_IMAGE=/local/scratch/ssd/mschnepf/slc6-pilot.img DEBUG [U=12089,P=114529] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_IMAGE DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_TIME=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Evaluating envar to clean: LC_NAME=de_DE.UTF-8 DEBUG [U=12089,P=114529] singularity_runtime_environment() Transposing environment DEBUG [U=12089,P=114529] singularity_runtime_environment() Converting envar 'SINGULARITYENV_HISTFILE' to 'HISTFILE' = '(null)' DEBUG [U=12089,P=114529] envar_set() Unsetting environment variable: HISTFILE DEBUG [U=12089,P=114529] singularity_runtime_environment() Converting envar 'SINGULARITYENV_PATH' to 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin' DEBUG [U=12089,P=114529] envar_set() Setting environment variable: 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin' DEBUG [U=12089,P=114529] singularity_priv_drop_perm() Called singularity_priv_drop_perm(void) DEBUG [U=12089,P=114529] singularity_priv_drop_perm() Escalating permissison so we can properly drop permission DEBUG [U=12089,P=114529] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114529] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114529] singularity_priv_drop_perm() Resetting supplementary groups DEBUG [U=0,P=114529] singularity_priv_drop_perm() Dropping to group ID '1200' DEBUG [U=0,P=114529] singularity_priv_drop_perm() Dropping real and effective privileges to GID = '1200' DEBUG [U=0,P=114529] singularity_priv_drop_perm() Dropping real and effective privileges to UID = '12089' DEBUG [U=12089,P=114529] singularity_priv_drop_perm() Confirming we have correct GID DEBUG [U=12089,P=114529] singularity_priv_drop_perm() Confirming we have correct UID DEBUG [U=12089,P=114529] singularity_priv_drop_perm() Setting NO_NEW_PRIVS to prevent future privilege escalations. DEBUG [U=12089,P=114529] singularity_priv_drop_perm() Finished dropping privileges DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'TARGET_PWD' DEBUG [U=12089,P=114529] main() Attempting to chdir to CWD: /usr/users/mschnepf DEBUG [U=12089,P=114529] envar_set() Setting environment variable: 'SINGULARITY_CONTAINER' = 'slc6-pilot.img' DEBUG [U=12089,P=114529] envar_set() Setting environment variable: 'SINGULARITY_NAME' = 'slc6-pilot.img' DEBUG [U=12089,P=114529] singularity_registry_get() Returning NULL on 'SHELL' DEBUG [U=12089,P=114529] envar_set() Unsetting environment variable: SINGULARITY_SHELL DEBUG [U=12089,P=114529] singularity_registry_get() Returning value from registry: 'COMMAND' = 'shell' DEBUG [U=12089,P=114529] singularity_registry_get() Returning value from registry: 'COMMAND' = 'shell' LOG [U=12089,P=114529] main() USER=mschnepf, IMAGE='slc6-pilot.img', COMMAND='shell' INFO [U=12089,P=114529] action_shell() Singularity: Invoking an interactive shell within container...

DEBUG [U=12089,P=114529] action_shell() Exec'ing /.singularity.d/actions/shell`

[gmkurtzer]

Hi @mschnepf,

In the debug output, it seems like Singularity is doing the right thing so perhaps I misunderstood. With overlay enabled, Singularity is not masking the /usr directory and everything is working properly?

Thanks!

[mschnepf]

Hi gmkurtzer, with enabled overlayFS (enable overlay = yes) the mount works fine. With disabled enable overlay = no overlay /usr/ is empty. Here is the log for disabled overlayFS:

`singularity --debug shell /local/scratch/ssd/mschnepf/slc6-pilot.img Enabling debugging Ending argument loop Singularity version: 2.3-dist Exec'ing: /usr/local/libexec/singularity/cli/shell.exec Evaluating args: '/local/scratch/ssd/mschnepf/slc6-pilot.img' VERBOSE [U=0,P=114254] message_init() Set messagelevel to: 5 VERBOSE [U=0,P=114254] singularity_config_parse() Initialize configuration file: /usr/local/etc/singularity/singularity.conf DEBUG [U=0,P=114254] singularity_config_parse() Starting parse of configuration file /usr/local/etc/singularity/singularity.conf VERBOSE [U=0,P=114254] singularity_config_parse() Got config key allow setuid = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key max loop devices = '256' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key allow pid ns = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key config passwd = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key config group = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key config resolv_conf = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key mount proc = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key mount sys = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key mount dev = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key mount home = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key mount tmp = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key mount hostfs = 'no' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key bind path = '/etc/localtime' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key bind path = '/etc/hosts' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key user bind control = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key enable overlay = 'no' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key mount slave = 'yes' VERBOSE [U=0,P=114254] singularity_config_parse() Got config key sessiondir max size = '16' DEBUG [U=0,P=114254] singularity_config_parse() Finished parsing configuration file '/usr/local/etc/singularity/singularity.conf' VERBOSE [U=0,P=114254] singularity_registry_init() Initializing Singularity Registry VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(libexecdir, /usr/local/libexec) = 0 VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'COMMAND' = 'shell' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(COMMAND, shell) = 0 VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'MESSAGELEVEL' = '5' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(MESSAGELEVEL, 5) = 0 VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'VERSION' = '2.3-dist' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(version, 2.3-dist) = 0 VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(localstatedir, /usr/local/var) = 0 VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(sysconfdir, /usr/local/etc) = 0 VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'BINDIR' = '/usr/local/bin' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(bindir, /usr/local/bin) = 0 VERBOSE [U=0,P=114254] singularity_registry_set() Adding value to registry: 'IMAGE' = '/local/scratch/ssd/mschnepf/slc6-pilot.img' DEBUG [U=0,P=114254] singularity_registry_set() Returning singularity_registry_set(IMAGE, /local/scratch/ssd/mschnepf/slc6-pilot.img) = 0 DEBUG [U=0,P=114254] singularity_registry_get() Returning NULL on 'HOME' DEBUG [U=0,P=114254] singularity_registry_get() Returning NULL on 'TARGET_UID' DEBUG [U=0,P=114254] singularity_registry_get() Returning NULL on 'TARGET_GID' DEBUG [U=0,P=114254] singularity_priv_init() Initializing user info DEBUG [U=0,P=114254] singularity_priv_init() Set the calling user's username to: mschnepf DEBUG [U=0,P=114254] singularity_priv_init() Marking uinfo structure as ready DEBUG [U=0,P=114254] singularity_priv_init() Obtaining home directory VERBOSE [U=0,P=114254] singularity_priv_init() Set home (via getpwuid()) to: /usr/users/mschnepf VERBOSE [U=0,P=114254] singularity_suid_init() Running SUID program workflow VERBOSE [U=0,P=114254] singularity_suid_init() Checking program has appropriate permissions VERBOSE [U=0,P=114254] singularity_suid_init() Checking configuration file is properly owned by root VERBOSE [U=0,P=114254] singularity_suid_init() Checking if singularity.conf allows us to run as suid DEBUG [U=0,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow setuid, yes) DEBUG [U=0,P=114254] singularity_config_get_value_impl() Returning configuration value allow setuid='yes' DEBUG [U=0,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow setuid, yes) = 1 DEBUG [U=0,P=114254] singularity_registry_get() Returning NULL on 'NOSUID' VERBOSE [U=0,P=114254] singularity_priv_userns() Invoking the user namespace DEBUG [U=0,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow user ns, yes) DEBUG [U=0,P=114254] singularity_config_get_value_impl() No configuration entry found for 'allow user ns'; returning default value 'yes' DEBUG [U=0,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow user ns, yes) = 1 VERBOSE [U=0,P=114254] singularity_priv_userns() Not virtualizing USER namespace: running as SUID DEBUG [U=0,P=114254] singularity_priv_userns() Returning singularity_priv_init(void) DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_ns() Calling: _singularity_runtime_ns_pid() DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow pid ns, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value allow pid ns='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow pid ns, yes) = 1 DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'UNSHARE_PID' VERBOSE [U=12089,P=114254] singularity_runtime_ns_pid() Not virtualizing PID namespace on user request DEBUG [U=12089,P=114254] singularity_runtime_ns() Calling: _singularity_runtime_ns_ipc() DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(allow ipc ns, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() No configuration entry found for 'allow ipc ns'; returning default value 'yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(allow ipc ns, yes) = 1 DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'UNSHARE_IPC' VERBOSE [U=12089,P=114254] singularity_runtime_ns_ipc() Not virtualizing IPC namespace on user request DEBUG [U=12089,P=114254] singularity_runtime_ns() Calling: _singularity_runtime_ns_mnt() DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount slave, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount slave='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount slave, yes) = 1 DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114254] singularity_runtime_ns_mnt() Virtualizing FS namespace DEBUG [U=0,P=114254] singularity_runtime_ns_mnt() Virtualizing mount namespace DEBUG [U=0,P=114254] singularity_runtime_ns_mnt() Making mounts slave DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_sessiondir() Setting sessiondir VERBOSE [U=12089,P=114254] singularity_sessiondir() Using session directory: /usr/local/var/singularity/mnt/session DEBUG [U=12089,P=114254] singularity_sessiondir() Checking for session directory: /usr/local/var/singularity/mnt/session DEBUG [U=12089,P=114254] singularity_sessiondir() Obtaining the default sessiondir size DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value sessiondir max size='16' DEBUG [U=12089,P=114254] singularity_sessiondir() Converted sessiondir size to: 16 DEBUG [U=12089,P=114254] singularity_sessiondir() Creating the sessiondir size mount option length DEBUG [U=12089,P=114254] singularity_sessiondir() Got size length of: 9 DEBUG [U=12089,P=114254] singularity_sessiondir() Creating the sessiondir size mount option string DEBUG [U=12089,P=114254] singularity_sessiondir() Checking to make sure the string was allocated correctly DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114254] singularity_sessiondir() Mounting sessiondir tmpfs: /usr/local/var/singularity/mnt/session DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID VERBOSE [U=12089,P=114254] singularity_registry_set() Adding value to registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114254] singularity_registry_set() Returning singularity_registry_set(SESSIONDIR, /usr/local/var/singularity/mnt/session) = 0 DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CLEANUPDIR' DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'NOSESSIONCLEANUP' DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'NOCLEANUP' DEBUG [U=12089,P=114254] singularity_cleanupd() Not running a cleanup thread, no 'SINGULARITY_CLEANUPDIR' defined DEBUG [U=12089,P=114254] singularity_registry_get() Returning value from registry: 'IMAGE' = '/local/scratch/ssd/mschnepf/slc6-pilot.img' DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'WRITABLE' DEBUG [U=12089,P=114254] singularity_config_get_value_impl() No configuration entry found for 'limit container owners'; returning default value 'NULL' DEBUG [U=12089,P=114254] singularity_config_get_value_impl() No configuration entry found for 'limit container paths'; returning default value 'NULL' DEBUG [U=12089,P=114254] singularity_image_open() Opening file descriptor to image: /local/scratch/ssd/mschnepf/slc6-pilot.img VERBOSE [U=12089,P=114254] singularity_image_check() Checking that file pointer is a Singularity image DEBUG [U=12089,P=114254] singularity_image_check() First line of image(fd=4): #!/usr/bin/env run-singularity

DEBUG [U=12089,P=114254] singularity_image_check() Checking if first line matches key VERBOSE [U=12089,P=114254] singularity_image_check() File is a valid Singularity image DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value max loop devices='256' DEBUG [U=12089,P=114254] singularity_image_bind() Entered singularity_image_bind() DEBUG [U=12089,P=114254] singularity_image_bind() Converting max_loop_devs_string to int: '256' DEBUG [U=12089,P=114254] singularity_image_bind() Converted max_loop_devs_string to int: '256' -> 256 DEBUG [U=12089,P=114254] singularity_image_bind() Checking if this image has been properly opened DEBUG [U=12089,P=114254] singularity_image_bind() Checking if image is valid file DEBUG [U=12089,P=114254] singularity_image_bind() Checking if image is already bound to a loop device DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'WRITABLE' DEBUG [U=12089,P=114254] singularity_image_bind() Setting loopdev open to: O_RDONLY DEBUG [U=12089,P=114254] singularity_image_bind() Setting LO_FLAGS_AUTOCLEAR DEBUG [U=12089,P=114254] singularity_image_bind() Calculating image offset VERBOSE [U=12089,P=114254] singularity_image_check() Checking that file pointer is a Singularity image DEBUG [U=12089,P=114254] singularity_image_check() First line of image(fd=4): #!/usr/bin/env run-singularity

DEBUG [U=12089,P=114254] singularity_image_check() Checking if first line matches key VERBOSE [U=12089,P=114254] singularity_image_check() File is a valid Singularity image VERBOSE [U=12089,P=114254] singularity_image_offset() Calculating image offset VERBOSE [U=12089,P=114254] singularity_image_offset() Found image at an offset of 31 bytes DEBUG [U=12089,P=114254] singularity_image_offset() Returning image_offset(image_fp) = 31 DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114254] singularity_image_bind() Finding next available loop device... VERBOSE [U=0,P=114254] singularity_image_bind() Found available loop device: /dev/loop0 DEBUG [U=0,P=114254] singularity_image_bind() Setting loop device flags DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID VERBOSE [U=12089,P=114254] singularity_image_bind() Using loop device: /dev/loop0 VERBOSE [U=12089,P=114254] singularity_runtime_rootfs() Set container directory to: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Checking for container directory DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] chk_mode() Checking exact mode (40755) on: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] chk_mode() Found appropriate mode on file: /usr/local/var/singularity/mnt/container VERBOSE [U=12089,P=114254] singularity_image_mount() Checking what kind of image we are mounting DEBUG [U=12089,P=114254] singularity_image_mount_squashfs_check() Image does not appear to be of type '.sqsh': /local/scratch/ssd/mschnepf/slc6-pilot.img DEBUG [U=12089,P=114254] singularity_image_mount_dir_check() Checking if image is a directory VERBOSE [U=12089,P=114254] singularity_image_mount_dir_check() Source path is not a directory: /local/scratch/ssd/mschnepf/slc6-pilot.img VERBOSE [U=12089,P=114254] singularity_image_mount() Attempting to mount as singularity image DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'WRITABLE' DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_image_mount_image_mount() Mounting /dev/loop0 in read/only to: /usr/local/var/singularity/mnt/container DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] action_ready() Checking if container is valid at: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_runtime_overlayfs() Checking if overlayfs should be used DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(enable overlay, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value enable overlay='no' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(enable overlay, yes) = 0 VERBOSE [U=12089,P=114254] singularity_runtime_overlayfs() Not enabling overlayFS via configuration DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container VERBOSE [U=12089,P=114254] singularity_runtime_mounts() Running all mount components DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount hostfs, no) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount hostfs='no' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount hostfs, no) = 0 DEBUG [U=12089,P=114254] singularity_runtime_mount_hostfs() Not mounting host file systems per configuration DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114254] singularity_runtime_mount_binds() Checking configuration file for 'bind path' VERBOSE [U=12089,P=114254] singularity_runtime_mount_binds() Found 'bind path' = /etc/localtime, /etc/localtime DEBUG [U=12089,P=114254] singularity_runtime_mount_binds() Checking if bind point is already mounted: /etc/localtime DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114254] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_binds() Binding '/etc/localtime' to '/usr/local/var/singularity/mnt/container//etc/localtime' DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID VERBOSE [U=12089,P=114254] singularity_runtime_mount_binds() Found 'bind path' = /etc/hosts, /etc/hosts DEBUG [U=12089,P=114254] singularity_runtime_mount_binds() Checking if bind point is already mounted: /etc/hosts DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114254] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_binds() Binding '/etc/hosts' to '/usr/local/var/singularity/mnt/container//etc/hosts' DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_runtime_mount_kernelfs() Checking configuration file for 'mount proc' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount proc, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount proc='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount proc, yes) = 1 DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'PIDNS_ENABLED' DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_kernelfs() Bind-mounting host /proc DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_mount_kernelfs() Checking configuration file for 'mount sys' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount sys, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount sys='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount sys, yes) = 1 DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_kernelfs() Mounting /sys DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount dev='yes' DEBUG [U=12089,P=114254] singularity_runtime_mount_dev() Checking configuration file for 'mount dev' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount dev, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount dev='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount dev, yes) = 1 DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_dev() Bind mounting /dev DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount home, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount home='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount home, yes) = 1 DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking that home directry is configured: /usr/users/mschnepf DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking if SINGULARITY_CONTAIN is set DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking if home directories are being influenced by user DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'HOME' DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking ownership of home directory source: /usr/users/mschnepf DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking to make sure home directory destination is a full path: /usr/users/mschnepf DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking if home directory is already mounted: /usr/users/mschnepf DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114254] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking if overlay is enabled DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'OVERLAYFS_ENABLED' DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Staging home directory DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking if sessiondir/tmpdir is set DEBUG [U=12089,P=114254] singularity_registry_get() Returning value from registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Identifying the base home directory: /usr/users/mschnepf DEBUG [U=12089,P=114254] basedir() Obtaining basedir for: /usr/users/mschnepf DEBUG [U=12089,P=114254] basedir() Iterating basedir: /usr/users/mschnepf DEBUG [U=12089,P=114254] basedir() Iterating basedir: /usr/users DEBUG [U=12089,P=114254] basedir() Iterating basedir: /usr DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Checking home directory base exists in container: /usr DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Creating temporary directory to stage home: /usr/local/var/singularity/mnt/session/usr/users/mschnepf DEBUG [U=12089,P=114254] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/session/usr DEBUG [U=12089,P=114254] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/session/usr/users DEBUG [U=12089,P=114254] s_mkpath() Creating directory: /usr/local/var/singularity/mnt/session/usr/users/mschnepf DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_home() Mounting home directory source to stage: /usr/users/mschnepf -> /usr/local/var/singularity/mnt/session/usr/users/mschnepf DEBUG [U=0,P=114254] singularity_runtime_mount_home() Remounting home directory with necessary options: /usr/local/var/singularity/mnt/session/usr/users/mschnepf DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Setting home directory source to: '/usr/local/var/singularity/mnt/session' + '/usr' DEBUG [U=12089,P=114254] singularity_runtime_mount_home() Setting home directory dest to: '/usr/local/var/singularity/mnt/container' + '/usr' DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_home() Mounting home directory source into container: /usr/local/var/singularity/mnt/session/usr -> /usr/local/var/singularity/mnt/container/usr DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] envar_set() Setting environment variable: 'HOME' = '/usr/users/mschnepf' DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_runtime_mount_userbinds() Checking for environment variable 'SINGULARITY_BINDPATH' DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'BINDPATH' DEBUG [U=12089,P=114254] singularity_runtime_mount_userbinds() No user bind mounts specified. DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(mount tmp, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value mount tmp='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(mount tmp, yes) = 1 DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114254] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_tmp() Mounting directory: /tmp DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] check_mounted() Opening /proc/mounts DEBUG [U=12089,P=114254] check_mounted() Iterating through /proc/mounts DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_mount_tmp() Mounting directory: /var/tmp DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_runtime_mount_scratch() Getting SINGULARITY_SCRATCHDIR from environment DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'SCRATCHDIR' DEBUG [U=12089,P=114254] singularity_runtime_mount_scratch() Not mounting scratch directory: Not requested DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_runtime_mount_cwd() Checking to see if we should mount current working directory DEBUG [U=12089,P=114254] singularity_runtime_mount_cwd() Getting current working directory DEBUG [U=12089,P=114254] singularity_runtime_mount_cwd() Checking if current directory exists in container DEBUG [U=12089,P=114254] file_devino() Called file_devino(/usr/users/mschnepf) DEBUG [U=12089,P=114254] file_devino() Returning file_devino(/usr/users/mschnepf) = 49.23068673 DEBUG [U=12089,P=114254] file_devino() Called file_devino(/usr/local/var/singularity/mnt/container/usr/users/mschnepf) DEBUG [U=12089,P=114254] file_devino() Returning file_devino(/usr/local/var/singularity/mnt/container/usr/users/mschnepf) = 49.23068673 DEBUG [U=12089,P=114254] singularity_runtime_mount_cwd() Checking if container's cwd == host's cwd VERBOSE [U=12089,P=114254] singularity_runtime_mount_cwd() Not mounting current directory: location already available within container DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container VERBOSE [U=12089,P=114254] singularity_runtime_files() Running file components DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_registry_get() Returning value from registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114254] singularity_runtime_files_passwd() Called singularity_file_passwd_create() DEBUG [U=12089,P=114254] singularity_runtime_files_passwd() Checking configuration option: 'config passwd' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(config passwd, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value config passwd='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(config passwd, yes) = 1 VERBOSE [U=12089,P=114254] singularity_runtime_files_passwd() Checking for template passwd file: /usr/local/var/singularity/mnt/container/etc/passwd VERBOSE [U=12089,P=114254] singularity_runtime_files_passwd() Creating template of /etc/passwd DEBUG [U=12089,P=114254] copy_file() Called copy_file(/usr/local/var/singularity/mnt/container/etc/passwd, /usr/local/var/singularity/mnt/session/passwd) DEBUG [U=12089,P=114254] copy_file() Opening source file: /usr/local/var/singularity/mnt/container/etc/passwd DEBUG [U=12089,P=114254] copy_file() Opening destination file: /usr/local/var/singularity/mnt/session/passwd DEBUG [U=12089,P=114254] copy_file() Calling fstat() on source file descriptor: 6 DEBUG [U=12089,P=114254] copy_file() Cloning permission string of source to dest DEBUG [U=12089,P=114254] copy_file() Copying file data... DEBUG [U=12089,P=114254] copy_file() Done copying data, closing file pointers DEBUG [U=12089,P=114254] copy_file() Returning copy_file(/usr/local/var/singularity/mnt/container/etc/passwd, /usr/local/var/singularity/mnt/session/passwd) = 0 VERBOSE [U=12089,P=114254] singularity_runtime_files_passwd() Creating template passwd file and appending user data: /usr/local/var/singularity/mnt/session/passwd DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] container_file_bind() Called file_bind(/usr/local/var/singularity/mnt/session/passwd, /etc/passwd() DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] container_file_bind() Binding file '/usr/local/var/singularity/mnt/session/passwd' to '/usr/local/var/singularity/mnt/container/etc/passwd' DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] envar_set() Setting environment variable: 'HOME' = '/usr/users/mschnepf' DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_registry_get() Returning value from registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114254] singularity_runtime_files_group() Called singularity_file_group_create() DEBUG [U=12089,P=114254] singularity_runtime_files_group() Checking configuration option: 'config group' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(config group, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value config group='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(config group, yes) = 1 VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Creating template of /etc/group for containment DEBUG [U=12089,P=114254] copy_file() Called copy_file(/usr/local/var/singularity/mnt/container/etc/group, /usr/local/var/singularity/mnt/session/group) DEBUG [U=12089,P=114254] copy_file() Opening source file: /usr/local/var/singularity/mnt/container/etc/group DEBUG [U=12089,P=114254] copy_file() Opening destination file: /usr/local/var/singularity/mnt/session/group DEBUG [U=12089,P=114254] copy_file() Calling fstat() on source file descriptor: 6 DEBUG [U=12089,P=114254] copy_file() Cloning permission string of source to dest DEBUG [U=12089,P=114254] copy_file() Copying file data... DEBUG [U=12089,P=114254] copy_file() Done copying data, closing file pointers DEBUG [U=12089,P=114254] copy_file() Returning copy_file(/usr/local/var/singularity/mnt/container/etc/group, /usr/local/var/singularity/mnt/session/group) = 0 VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Updating group file with user info DEBUG [U=12089,P=114254] singularity_runtime_files_group() Getting supplementary group info VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Found supplementary group membership in: 132 VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Adding user's supplementary group ('docker') info to template group file VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Found supplementary group membership in: 500 VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Adding user's supplementary group ('printer') info to template group file DEBUG [U=12089,P=114254] singularity_runtime_files_group() Skipping duplicate supplementary group VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Found supplementary group membership in: 1900 VERBOSE [U=12089,P=114254] singularity_runtime_files_group() Adding user's supplementary group ('ekpadmin') info to template group file DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] container_file_bind() Called file_bind(/usr/local/var/singularity/mnt/session/group, /etc/group() DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] container_file_bind() Binding file '/usr/local/var/singularity/mnt/session/group' to '/usr/local/var/singularity/mnt/container/etc/group' DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_files_resolvconf() Checking configuration option DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Called singularity_config_get_bool(config resolv_conf, yes) DEBUG [U=12089,P=114254] singularity_config_get_value_impl() Returning configuration value config resolv_conf='yes' DEBUG [U=12089,P=114254] singularity_config_get_bool_char_impl() Return singularity_config_get_bool(config resolv_conf, yes) = 1 DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] container_file_bind() Called file_bind(/etc/resolv.conf, /etc/resolv.conf() DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] container_file_bind() Binding file '/etc/resolv.conf' to '/usr/local/var/singularity/mnt/container/etc/resolv.conf' DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_registry_get() Returning value from registry: 'SESSIONDIR' = '/usr/local/var/singularity/mnt/session' DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CONTAINLIBS' DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container VERBOSE [U=12089,P=114254] singularity_runtime_enter() Containing all rootfs components DEBUG [U=12089,P=114254] singularity_runtime_rootfs() Returning container_directory: /usr/local/var/singularity/mnt/container DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. VERBOSE [U=0,P=114254] singularity_runtime_enter_chroot() Entering container file system root: /usr/local/var/singularity/mnt/container DEBUG [U=0,P=114254] singularity_priv_drop() Dropping privileges to UID=12089, GID=1200 (4 supplementary GIDs) DEBUG [U=0,P=114254] singularity_priv_drop() Restoring supplementary groups DEBUG [U=12089,P=114254] singularity_priv_drop() Confirming we have correct UID/GID DEBUG [U=12089,P=114254] singularity_runtime_enter_chroot() Changing dir to '/' within the new root DEBUG [U=12089,P=114254] singularity_runtime_environment() Cloning environment DEBUG [U=12089,P=114254] singularity_runtime_environment() Counted 37 environment elements DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CLEANENV' DEBUG [U=12089,P=114254] singularity_runtime_environment() Cleaning environment DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_PAPER=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_ADDRESS=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: XDG_SESSION_ID=939 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_MONETARY=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: TERM=xterm-256color DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SHELL=/bin/bash DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SSH_CLIENT=172.23.242.246 33654 22 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_NUMERIC=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SSH_TTY=/dev/pts/8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_libexecdir=/usr/local/libexec DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_libexecdir DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_COMMAND=shell DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_COMMAND DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: USER=mschnepf DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_MESSAGELEVEL=5 DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_MESSAGELEVEL DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_TELEPHONE=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SSH_AUTH_SOCK=/tmp/ssh-xBnNIKo5je/agent.114196 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_version=2.3-dist DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_version DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: MAIL=/var/mail/mschnepf DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_localstatedir=/usr/local/var DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_localstatedir DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: QT_QPA_PLATFORMTHEME=appmenu-qt5 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_IDENTIFICATION=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: PWD=/usr/users/mschnepf DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITYENV_HISTFILE= DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITYENV_PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LANG=en_US.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_MEASUREMENT=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SHLVL=1 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: HOME=/usr/users/mschnepf DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_sysconfdir=/usr/local/etc DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_sysconfdir DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LOGNAME=mschnepf DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_bindir=/usr/local/bin DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_bindir DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SSH_CONNECTION=172.23.242.246 33654 129.13.102.147 22 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: XDG_RUNTIME_DIR=/run/user/12089 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: DISPLAY=localhost:10.0 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: SINGULARITY_IMAGE=/local/scratch/ssd/mschnepf/slc6-pilot.img DEBUG [U=12089,P=114254] singularity_runtime_environment() Unsetting environment variable: SINGULARITY_IMAGE DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_TIME=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Evaluating envar to clean: LC_NAME=de_DE.UTF-8 DEBUG [U=12089,P=114254] singularity_runtime_environment() Transposing environment DEBUG [U=12089,P=114254] singularity_runtime_environment() Converting envar 'SINGULARITYENV_HISTFILE' to 'HISTFILE' = '(null)' DEBUG [U=12089,P=114254] envar_set() Unsetting environment variable: HISTFILE DEBUG [U=12089,P=114254] singularity_runtime_environment() Converting envar 'SINGULARITYENV_PATH' to 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin' DEBUG [U=12089,P=114254] envar_set() Setting environment variable: 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin' DEBUG [U=12089,P=114254] singularity_priv_drop_perm() Called singularity_priv_drop_perm(void) DEBUG [U=12089,P=114254] singularity_priv_drop_perm() Escalating permissison so we can properly drop permission DEBUG [U=12089,P=114254] singularity_priv_escalate() Temporarily escalating privileges (U=12089) DEBUG [U=0,P=114254] singularity_priv_escalate() Clearing supplementary GIDs. DEBUG [U=0,P=114254] singularity_priv_drop_perm() Resetting supplementary groups DEBUG [U=0,P=114254] singularity_priv_drop_perm() Dropping to group ID '1200' DEBUG [U=0,P=114254] singularity_priv_drop_perm() Dropping real and effective privileges to GID = '1200' DEBUG [U=0,P=114254] singularity_priv_drop_perm() Dropping real and effective privileges to UID = '12089' DEBUG [U=12089,P=114254] singularity_priv_drop_perm() Confirming we have correct GID DEBUG [U=12089,P=114254] singularity_priv_drop_perm() Confirming we have correct UID DEBUG [U=12089,P=114254] singularity_priv_drop_perm() Setting NO_NEW_PRIVS to prevent future privilege escalations. DEBUG [U=12089,P=114254] singularity_priv_drop_perm() Finished dropping privileges DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'CONTAIN' DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'TARGET_PWD' DEBUG [U=12089,P=114254] main() Attempting to chdir to CWD: /usr/users/mschnepf DEBUG [U=12089,P=114254] envar_set() Setting environment variable: 'SINGULARITY_CONTAINER' = 'slc6-pilot.img' DEBUG [U=12089,P=114254] envar_set() Setting environment variable: 'SINGULARITY_NAME' = 'slc6-pilot.img' DEBUG [U=12089,P=114254] singularity_registry_get() Returning NULL on 'SHELL' DEBUG [U=12089,P=114254] envar_set() Unsetting environment variable: SINGULARITY_SHELL DEBUG [U=12089,P=114254] singularity_registry_get() Returning value from registry: 'COMMAND' = 'shell' DEBUG [U=12089,P=114254] singularity_registry_get() Returning value from registry: 'COMMAND' = 'shell' LOG [U=12089,P=114254] main() USER=mschnepf, IMAGE='slc6-pilot.img', COMMAND='shell' INFO [U=12089,P=114254] action_shell() Singularity: Invoking an interactive shell within container...

DEBUG [U=12089,P=114254] action_shell() Exec'ing /.singularity.d/actions/shell `

[gmkurtzer]

Yep, this is expected behavior.

[mschnepf]

I would expect that with disabled overlay fs singularity do not mount the home directory instead of "erase" directories in the container.

However, I can mount my home from host (/usr/users/mschnepf) in the container (/usr/users/mschnepf) with disabled overlay fs ( enable overlay = no ). With the command singularity shell -B /usr/users/mschnepf:/usr/users/mschnepf image.img and the option mount home = no I have access to /usr/bin, /usr/lib, /usr/... and my home /usr/users/mschnepf inside the container. So I think, because this case works it should also work with the option mount home = no

[gmkurtzer]

Hi @mschnepf,

I'm not sure I follow exactly what you are getting at, but for the current implementation of the home mount Singularity module, this is expected. I do think that it could use some optimization such that this gets handled smarter and I'll add that to the next big milestone.

Thanks!

[draeath]

I've observed that the home mount (the default in /home or user-specified with -H) causes the parent directory in-container to be replaced with a tmpfs. This masks any content that might already exist there.

Is there some reason the home module mounts behave differently than the additional bind mounts requested at runtime with -B?

You can work around this problem by pre-creating a unique parent directory for home to go under, inside the container, but this may require editing/extending an existing (eg, docker) image.

This is especially irritating when working with docker images that assume /home will be left alone, and that have important content there (eg, the biobakery project's docker images use the linuxbrew docker image as a base, so everything of importance is in /home/linuxbrew/.linuxbrew - which is inaccessible unless one puts home somewhere else with -H - of which there are no good directories to do this with in said image.)

[jscook2345]

It looks like this is resolved. I opened a new issue for the last comment, @draeath. #2278