remaining .singularity-runtime folder in $SINGULARITY_LOCALCACHEDIR

Quanteek commented 7 years ago

Hey,

First of all thank you for this amazing job.

I am a beginner with Singularity so might not be a bug but expected behavior (but did not find something on that in the doc). I am using an HPC system with a frontend and computation nodes managed with PBS.

Version of Singularity:

2.3

Expected behavior

At the end of my PBS script I want to have zero elements of singularity on the disk since it takes a lot of space (2Go per run). I am using SINGULARITY_LOCALCACHEDIR on the current dir since our /tmp is quite small for each node.

Actual behavior

.singularity-runtime folder is still available after run.

Steps to reproduce behavior

You can execute this script if you have PBS on your machine :

#PBS -S /bin/bash                                                                                                                             
#PBS -N singularity                                                                                                                           
#PBS -j oe                                                                                                                                    
#PBS -o logs/                                                                                                                                 
#PBS -l walltime=12:00:00                                                                                                                     
#PBS -l select=1:ncpus=4:mem=20gb                                                                                                             
#PBS -q haswellq                                                                                                                              

cd $PBS_O_WORKDIR                                                                                                                             

export SINGULARITY_LOCALCACHEDIR=$PWD                                                                                                         

echo "pulling"                                                                                                                                
singularity pull docker://quanteek/docker-r-sis                                                                                               

echo "running"                                                                                                                                
singularity exec docker://quanteek/docker-r-sis /home/user01/run.sh p.txt y.txt  

#when the job quits there is still a folder in $PWD which can be heavy in size which can prevent multiple run.

Thank you for your help.

gmkurtzer commented 7 years ago

Hi @Quanteek,

I can't seem to reproduce this behavior:

$ singularity --version
2.3-master.gda9cd20
$ SINGULARITY_LOCALCACHEDIR=`pwd` singularity exec docker://centos cat /etc/redhat-release; ls -d .singularity_runtime*
Docker image path: index.docker.io/library/centos:latest
Cache folder set to /home/gmk/.singularity/docker
[1/1] |===================================| 100.0% 
Creating container runtime...
CentOS Linux release 7.3.1611 (Core) 
ls: cannot access .singularity_runtime*: No such file or directory

Quanteek commented 7 years ago

Hey @gmkurtzer

Thank you for your reply. Sorry I have not been clear.

If I do what you do in a PBS script then it ends like you :

cat logs/211501.OU 
Docker image path: index.docker.io/library/centos:latest
Cache folder set to /gpfs/home/xxx/.singularity/docker
Creating container runtime...
CentOS Linux release 7.3.1611 (Core) 
ls: cannot access .singularity_runtime*: No such file or directory

but if I do lsin the directory after completion of the job I have :

[xxx@xxx singularity]$ ls -al
total 1318912
drwx------  4 xxx xxx       4096 Jun  7 23:18 .
drwxr-xr-x 15 xxx xxx       4096 Jun  7 18:22 ..
-rwxr-xr-x  1 xxx xxx 1347420192 Jun  7 16:15 docker-r-sis.img
drwxr-xr-x  2 xxx xxx       4096 Jun  7 23:18 logs
-rw-r--r--  1 xxx xxx        390 Jun  7 23:18 script2.pbs
-rw-r--r--  1 xxx xxx        430 Jun  7 23:17 script2.pbs~
-rw-r--r--  1 xxx xxx        430 Jun  7 17:52 script.pbs
-rw-r--r--  1 xxx xxx        487 Jun  7 17:45 script.pbs~
drwx------  3 xxx xxx       4096 Jun  7 23:18 .singularity-runtime.jwR7Fi38
-rw-r--r--  1 xxx xxx          0 Jun  7 17:53 y.txt

Thus the singularity-runtime.jwR7Fi38 folder has not been deleted at the end.

Below is the PBS script :

#PBS -S /bin/bash
#PBS -N singularity
#PBS -j oe
#PBS -o logs/
#PBS -l walltime=12:00:00
#PBS -l select=1:ncpus=4:mem=20gb
#PBS -q haswellq

cd $PBS_O_WORKDIR

#export SINGULARITY_LOCALCACHEDIR=$PWD

SINGULARITY_LOCALCACHEDIR=`pwd` singularity exec docker://centos cat /etc/redhat-release; ls -d .singularity_runtime*

Quanteek commented 7 years ago

I have checked the PBS output and I have an exit status to 2. Might be a clue that something went wrong with a command.

EDIT : after checking $?, it is the ls that return 2 but singularity returns 0.

Might be a problem with the filesystem ?

gmkurtzer commented 7 years ago

Hrmm.. very interesting. Can you rerun with the --debug option, and paste all of the output?

Thanks!

Quanteek commented 7 years ago

Enabling debugging
Ending argument loop
Singularity version: 2.3-dist
Exec'ing: /gpfs/opt/tools/singularity/2.3/libexec/singularity/cli/exec.exec
Evaluating args: 'docker://centos cat /etc/redhat-release'
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos
VERBOSE2 SINGULARITY_METADATA_FOLDER found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /home/my_user/.singularity
VERBOSE2 SINGULARITY_ENVIRONMENT found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos/.singularity.d/environment
VERBOSE2 SINGULARITY_RUNSCRIPT found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos/singularity
VERBOSE2 SINGULARITY_TESTFILE found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos/.singularity.d/Singularity
VERBOSE2 SINGULARITY_ENVBASE found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as /gpfs/workdir/my_user/Code/singularity
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layers.cjnQiYDr
VERBOSE2 SINGULARITY_PYTHREADS found as 9
VERBOSE2 SINGULARITY_CONTAINER found as docker://centos
DEBUG Found uri docker://
DEBUG 
*** STARTING DOCKER IMPORT PYTHON  ****
DEBUG Docker layers and (env,labels,runscript) will be written to: /tmp/.singularity-layers.cjnQiYDr
VERBOSE2 SINGULARITY_DOCKER_USERNAME not defined (None)
VERBOSE2 SINGULARITY_DOCKER_PASSWORD found
DEBUG Starting Docker IMPORT, includes environment, runscript, and metadata.
VERBOSE Docker image: centos
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: index.docker.io
VERBOSE Namespace: library
VERBOSE Repo Name: centos
VERBOSE Repo Tag: latest
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/library/centos/tags/list
DEBUG GET https://index.docker.io/v2/library/centos/tags/list
DEBUG Http Error with code 401
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/centos:pull
DEBUG Headers found: Content-Type,Authorization,Accept
VERBOSE3 Response on obtaining token is None.
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/centos/manifests/latest
DEBUG GET https://index.docker.io/v2/library/centos/manifests/latest
DEBUG Image manifest version 2.2 found.
DEBUG Adding digest sha256:d5e46245fe40c2d1ab72bfe328de28549b605b2587ab2fa8715f54e3e2de9c5d
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/centos/manifests/latest
DEBUG GET https://index.docker.io/v2/library/centos/manifests/latest
DEBUG Using 9 workers for multiprocess.
VERBOSE3 Found Docker command (Entrypoint) None
VERBOSE3 Found Docker command (Cmd) /bin/bash
VERBOSE3 Adding Docker CMD as Singularity runscript...
DEBUG /bin/bash
VERBOSE3 Found Docker command (Env) PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE3 Found Docker container environment!
VERBOSE3 Adding Docker environment to metadata tar
DEBUG Found template for tarinfo
VERBOSE3 Found Docker command (Labels) {u'build-date': u'20170605', u'vendor': u'CentOS', u'name': u'CentOS Base Image', u'license': u'GPLv2'}
VERBOSE3 Found Docker container labels!
VERBOSE3 Adding Docker labels to metadata tar
DEBUG Found template for tarinfo
VERBOSE3 Adding Docker runscript to metadata tar
DEBUG Found template for tarinfo
DEBUG metadata file /gpfs/home/my_user/.singularity/metadata/sha256:6b8bbe197a20c88d065c265cf6f6f8b4e3695f104d1f47f01a1298b3566f27fe.tar.gz already exists, will over-write.
VERBOSE2 Tar file with Docker env and labels: /gpfs/home/my_user/.singularity/metadata/sha256:6b8bbe197a20c88d065c265cf6f6f8b4e3695f104d1f47f01a1298b3566f27fe.tar.gz
VERBOSE3 Writing Docker layers files to /tmp/.singularity-layers.cjnQiYDr
VERBOSE2 Writing file /tmp/.singularity-layers.cjnQiYDr with mode w.
VERBOSE2 Writing file /tmp/.singularity-layers.cjnQiYDr with mode a.
DEBUG *** FINISHING DOCKER IMPORT PYTHON PORTION ****
Docker image path: index.docker.io/library/centos:latest
Cache folder set to /gpfs/home/my_user/.singularity/docker
Creating container runtime...
Importing: base Singularity environment
Exploding layer: sha256:d5e46245fe40c2d1ab72bfe328de28549b605b2587ab2fa8715f54e3e2de9c5d.tar.gz
Exploding layer: sha256:6b8bbe197a20c88d065c265cf6f6f8b4e3695f104d1f47f01a1298b3566f27fe.tar.gz
VERBOSE [U=0,P=27606]      message_init()                            Set messagelevel to: 5
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Initialize configuration file: /gpfs/opt/tools/singularity/2.3/etc/singularity/singularity.conf
[0mDEBUG   [U=0,P=27606]      singularity_config_parse()                Starting parse of configuration file /gpfs/opt/tools/singularity/2.3/etc/singularity/singularity.conf
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key allow setuid = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key max loop devices = '256'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key allow pid ns = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key config passwd = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key config group = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key config resolv_conf = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key mount proc = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key mount sys = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key mount dev = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key mount home = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key mount tmp = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key mount hostfs = 'no'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key bind path = '/etc/localtime'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key bind path = '/etc/hosts'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key user bind control = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key enable overlay = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key mount slave = 'yes'
[0mVERBOSE [U=0,P=27606]      singularity_config_parse()                Got config key sessiondir max size = '16'
[0mDEBUG   [U=0,P=27606]      singularity_config_parse()                Finished parsing configuration file '/gpfs/opt/tools/singularity/2.3/etc/singularity/singularity.conf'
[0mVERBOSE [U=0,P=27606]      singularity_registry_init()               Initializing Singularity Registry
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/gpfs/opt/tools/singularity/2.3/libexec'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(libexecdir, /gpfs/opt/tools/singularity/2.3/libexec) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'COMMAND' = 'exec'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(COMMAND, exec) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'ROOTFS' = '/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(ROOTFS, /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'VERSION' = '2.3-dist'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(version, 2.3-dist) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/gpfs/opt/tools/singularity/2.3/var'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(localstatedir, /gpfs/opt/tools/singularity/2.3/var) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'LOCALCACHEDIR' = '/workdir/my_user/Code/singularity'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(LOCALCACHEDIR, /workdir/my_user/Code/singularity) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'CONTENTS' = '/tmp/.singularity-layers.cjnQiYDr'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(CONTENTS, /tmp/.singularity-layers.cjnQiYDr) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/gpfs/opt/tools/singularity/2.3/etc'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /gpfs/opt/tools/singularity/2.3/etc) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'BINDIR' = '/gpfs/opt/tools/singularity/2.3/bin'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(bindir, /gpfs/opt/tools/singularity/2.3/bin) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(CLEANUPDIR, /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'CONTAINER' = 'docker://centos'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(CONTAINER, docker://centos) = 0
[0mVERBOSE [U=0,P=27606]      singularity_registry_set()                Adding value to registry: 'IMAGE' = '/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos'
[0mDEBUG   [U=0,P=27606]      singularity_registry_set()                Returning singularity_registry_set(IMAGE, /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos) = 0
[0mDEBUG   [U=0,P=27606]      singularity_registry_get()                Returning NULL on 'HOME'
[0mDEBUG   [U=0,P=27606]      singularity_registry_get()                Returning NULL on 'TARGET_UID'
[0mDEBUG   [U=0,P=27606]      singularity_registry_get()                Returning NULL on 'TARGET_GID'
[0mDEBUG   [U=0,P=27606]      singularity_priv_init()                   Initializing user info
[0mDEBUG   [U=0,P=27606]      singularity_priv_init()                   Set the calling user's username to: my_user
[0mDEBUG   [U=0,P=27606]      singularity_priv_init()                   Marking uinfo structure as ready
[0mDEBUG   [U=0,P=27606]      singularity_priv_init()                   Obtaining home directory
[0mVERBOSE [U=0,P=27606]      singularity_priv_init()                   Set home (via getpwuid()) to: /home/my_user
[0mVERBOSE [U=0,P=27606]      singularity_suid_init()                   Running SUID program workflow
[0mVERBOSE [U=0,P=27606]      singularity_suid_init()                   Checking program has appropriate permissions
[0mVERBOSE [U=0,P=27606]      singularity_suid_init()                   Checking configuration file is properly owned by root
[0mVERBOSE [U=0,P=27606]      singularity_suid_init()                   Checking if singularity.conf allows us to run as suid
[0mDEBUG   [U=0,P=27606]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow setuid, yes)
[0mDEBUG   [U=0,P=27606]      singularity_config_get_value_impl()       Returning configuration value allow setuid='yes'
[0mDEBUG   [U=0,P=27606]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow setuid, yes) = 1
[0mDEBUG   [U=0,P=27606]      singularity_registry_get()                Returning NULL on 'NOSUID'
[0mVERBOSE [U=0,P=27606]      singularity_priv_userns()                 Invoking the user namespace
[0mDEBUG   [U=0,P=27606]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
[0mDEBUG   [U=0,P=27606]      singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
[0mDEBUG   [U=0,P=27606]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
[0mVERBOSE [U=0,P=27606]      singularity_priv_userns()                 Not virtualizing USER namespace: running as SUID
[0mDEBUG   [U=0,P=27606]      singularity_priv_userns()                 Returning singularity_priv_init(void)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_ns()                  Calling: _singularity_runtime_ns_pid()
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow pid ns, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value allow pid ns='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow pid ns, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'UNSHARE_PID'
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_ns_pid()              Not virtualizing PID namespace on user request
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_ns()                  Calling: _singularity_runtime_ns_ipc()
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow ipc ns, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       No configuration entry found for 'allow ipc ns'; returning default value 'yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow ipc ns, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'UNSHARE_IPC'
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_ns_ipc()              Not virtualizing IPC namespace on user request
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_ns()                  Calling: _singularity_runtime_ns_mnt()
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount slave, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount slave='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount slave, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      singularity_runtime_ns_mnt()              Virtualizing FS namespace
[0mDEBUG   [U=0,P=27606]      singularity_runtime_ns_mnt()              Virtualizing mount namespace
[0mDEBUG   [U=0,P=27606]      singularity_runtime_ns_mnt()              Making mounts slave
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Setting sessiondir
[0mVERBOSE [U=1003,P=27606]   singularity_sessiondir()                  Using session directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Checking for session directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Obtaining the default sessiondir size
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value sessiondir max size='16'
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Converted sessiondir size to: 16
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Creating the sessiondir size mount option length
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Got size length of: 9
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Creating the sessiondir size mount option string
[0mDEBUG   [U=1003,P=27606]   singularity_sessiondir()                  Checking to make sure the string was allocated correctly
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      singularity_sessiondir()                  Mounting sessiondir tmpfs: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mVERBOSE [U=1003,P=27606]   singularity_registry_set()                Adding value to registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_set()                Returning singularity_registry_set(SESSIONDIR, /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session) = 0
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'NOSESSIONCLEANUP'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'NOCLEANUP'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session'
[0mDEBUG   [U=1003,P=27606]   singularity_cleanupd()                    Creating new cleanup trigger file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/cleanup_trigger
[0mDEBUG   [U=1003,P=27606]   singularity_cleanupd()                    Opening cleanup trigger file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/cleanup_trigger
[0mDEBUG   [U=1003,P=27606]   singularity_cleanupd()                    Gaining an exclusive flock on FD 4
[0mVERBOSE [U=1003,P=5667]    singularity_cleanupd()                    Exec'ing cleanupd thread: /gpfs/opt/tools/singularity/2.3/libexec/singularity/bin/cleanupd
[0mDEBUG   [U=1003,P=5667]    envar_set()                               Setting environment variable: 'SINGULARITY_CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4'
[0mDEBUG   [U=1003,P=5667]    envar_set()                               Setting environment variable: 'SINGULARITY_CLEANUPTRIGGER' = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/cleanup_trigger'
[0mVERBOSE [U=1003,P=5667]    message_init()                            Set messagelevel to: 5
[0mDEBUG   [U=1003,P=5667]    envar_path()                              Checking environment variable is valid path: 'SINGULARITY_CLEANUPDIR'
[0mVERBOSE [U=1003,P=5667]    envar_get()                               Checking input from environment: 'SINGULARITY_CLEANUPDIR'
[0mDEBUG   [U=1003,P=5667]    envar_get()                               Checking environment variable is defined: SINGULARITY_CLEANUPDIR
[0mDEBUG   [U=1003,P=5667]    envar_get()                               Checking environment variable length (<= 4096): SINGULARITY_CLEANUPDIR
[0mDEBUG   [U=1003,P=5667]    envar_get()                               Checking environment variable has allowed characters: SINGULARITY_CLEANUPDIR
[0mVERBOSE [U=1003,P=5667]    envar_get()                               Obtained input from environment 'SINGULARITY_CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4'
[0mDEBUG   [U=1003,P=5667]    envar_path()                              Checking environment variable is valid path: 'SINGULARITY_CLEANUPTRIGGER'
[0mVERBOSE [U=1003,P=5667]    envar_get()                               Checking input from environment: 'SINGULARITY_CLEANUPTRIGGER'
[0mDEBUG   [U=1003,P=5667]    envar_get()                               Checking environment variable is defined: SINGULARITY_CLEANUPTRIGGER
[0mDEBUG   [U=1003,P=5667]    envar_get()                               Checking environment variable length (<= 4096): SINGULARITY_CLEANUPTRIGGER
[0mDEBUG   [U=1003,P=5667]    envar_get()                               Checking environment variable has allowed characters: SINGULARITY_CLEANUPTRIGGER
[0mVERBOSE [U=1003,P=5667]    envar_get()                               Obtained input from environment 'SINGULARITY_CLEANUPTRIGGER' = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/cleanup_trigger'
[0mDEBUG   [U=1003,P=5667]    main()                                    Starting cleanup process
[0mDEBUG   [U=1003,P=5667]    main()                                    Opening cleanup trigger file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/cleanup_trigger
[0mDEBUG   [U=1003,P=5667]    main()                                    Checking to see if we need to daemonize
[0mVERBOSE [U=1003,P=5667]    main()                                    Daemonizing cleandir cleanup process
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'IMAGE' = '/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'WRITABLE'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       No configuration entry found for 'limit container owners'; returning default value 'NULL'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       No configuration entry found for 'limit container paths'; returning default value 'NULL'
[0mDEBUG   [U=1003,P=27606]   singularity_image_open()                  Opening file descriptor to image: /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos
[0mVERBOSE [U=1003,P=27606]   singularity_image_check()                 Image is a directory, returning retval=1: /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value max loop devices='256'
[0mDEBUG   [U=1003,P=27606]   singularity_image_bind()                  Entered singularity_image_bind()
[0mDEBUG   [U=1003,P=27606]   singularity_image_bind()                  Converting max_loop_devs_string to int: '256'
[0mDEBUG   [U=1003,P=27606]   singularity_image_bind()                  Converted max_loop_devs_string to int: '256' -> 256
[0mDEBUG   [U=1003,P=27606]   singularity_image_bind()                  Checking if this image has been properly opened
[0mDEBUG   [U=1003,P=27606]   singularity_image_bind()                  Checking if image is valid file
[0mVERBOSE [U=1003,P=27606]   singularity_image_bind()                  Skipping image bind, container is not a file
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_rootfs()              Set container directory to: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Checking for container directory
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mDEBUG   [U=1003,P=27606]   chk_mode()                                Checking exact mode (40755) on: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mDEBUG   [U=1003,P=27606]   chk_mode()                                Found appropriate mode on file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mVERBOSE [U=1003,P=27606]   singularity_image_mount()                 Checking what kind of image we are mounting
[0mDEBUG   [U=1003,P=27606]   singularity_image_mount_squashfs_check()  Image does not appear to be of type '.sqsh': /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos
[0mDEBUG   [U=1003,P=27606]   singularity_image_mount_dir_check()       Checking if image is a directory
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      singularity_image_mount_dir_mount()       Mounting container directory /workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos->/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mDEBUG   [U=1003,P=27606]   action_ready()                            Checking if container is valid at: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_overlayfs()           Checking if overlayfs should be used
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(enable overlay, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value enable overlay='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(enable overlay, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'DISABLE_OVERLAYFS'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'WRITABLE'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_overlayfs()           OverlayFS enabled by host build
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_overlayfs()           Setting up overlay mount options
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_overlayfs()           Checking for existance of overlay directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      singularity_runtime_overlayfs()           Mounting overlay tmpfs: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay
[0mDEBUG   [U=0,P=27606]      singularity_runtime_overlayfs()           Creating upper overlay directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/upper
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/upper
[0mDEBUG   [U=0,P=27606]      singularity_runtime_overlayfs()           Creating overlay work directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/work
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/work
[0mDEBUG   [U=0,P=27606]      singularity_runtime_overlayfs()           Creating overlay_final directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mVERBOSE [U=0,P=27606]      singularity_runtime_overlayfs()           Mounting overlay with options: lowerdir=/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/container,upperdir=/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/upper,workdir=/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/work
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mVERBOSE [U=1003,P=27606]   singularity_registry_set()                Adding value to registry: 'OVERLAYFS_ENABLED' = '1'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_set()                Returning singularity_registry_set(OVERLAYFS_ENABLED, 1) = 0
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_overlayfs()           Updating the containerdir to: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Setting container_directory = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_mounts()              Running all mount components
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount hostfs, no)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount hostfs='no'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount hostfs, no) = 0
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_hostfs()        Not mounting host file systems per configuration
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CONTAIN'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_binds()         Checking configuration file for 'bind path'
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_mount_binds()         Found 'bind path' = /etc/localtime, /etc/localtime
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_binds()         Checking if bind point is already mounted: /etc/localtime
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Opening /proc/mounts
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Iterating through /proc/mounts
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_binds()         Binding '/etc/localtime' to '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final//etc/localtime'
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_mount_binds()         Found 'bind path' = /etc/hosts, /etc/hosts
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_binds()         Checking if bind point is already mounted: /etc/hosts
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Opening /proc/mounts
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Iterating through /proc/mounts
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_binds()         Binding '/etc/hosts' to '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final//etc/hosts'
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount proc'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount proc, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount proc='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount proc, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'PIDNS_ENABLED'
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_kernelfs()      Bind-mounting host /proc
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount sys'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount sys, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount sys='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount sys, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_kernelfs()      Mounting /sys
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CONTAIN'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_dev()           Checking configuration file for 'mount dev'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount dev, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount dev, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_dev()           Bind mounting /dev
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount home, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount home='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount home, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Checking that home directry is configured: /home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Checking if SINGULARITY_CONTAIN is set
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CONTAIN'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Checking if home directories are being influenced by user
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'HOME'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Checking ownership of home directory source: /home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Checking to make sure home directory destination is a full path: /home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Checking if home directory is already mounted: /home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Opening /proc/mounts
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Iterating through /proc/mounts
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Checking if overlay is enabled
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'OVERLAYFS_ENABLED' = '1'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Binding home directory direct (no staging)
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Setting home directory source to: '/home/my_user'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Setting home directory dest to: '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final' + '/home/my_user'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Creating home directry within container: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/home/my_user
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_home()          Created home directory within the container: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_home()          Mounting home directory source into container: /home/my_user -> /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/home/my_user
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   envar_set()                               Setting environment variable: 'HOME' = '/home/my_user'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_userbinds()     Checking for environment variable 'SINGULARITY_BINDPATH'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'BINDPATH'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_userbinds()     No user bind mounts specified.
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount tmp, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value mount tmp='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount tmp, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CONTAIN'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Opening /proc/mounts
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Iterating through /proc/mounts
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_tmp()           Mounting directory: /tmp
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Opening /proc/mounts
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Iterating through /proc/mounts
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_mount_tmp()           Mounting directory: /var/tmp
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_scratch()       Getting SINGULARITY_SCRATCHDIR from environment
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'SCRATCHDIR'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_scratch()       Not mounting scratch directory: Not requested
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Checking to see if we should mount current working directory
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Getting current working directory
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Checking if current directory exists in container
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Container does not have the directory: /workdir/my_user/Code/singularity
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Checking for contain option
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CONTAIN'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Checking if CWD is already mounted: /workdir/my_user/Code/singularity
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Opening /proc/mounts
[0mDEBUG   [U=1003,P=27606]   check_mounted()                           Iterating through /proc/mounts
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Checking if cwd is in an operating system directory
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Checking if overlay is enabled
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'OVERLAYFS_ENABLED' = '1'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Checking configuration file for 'user bind control'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(user bind control, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value user bind control='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(user bind control, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_mount_cwd()           Creating current directory within container: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir/my_user/Code/singularity
[0mDEBUG   [U=1003,P=27606]   s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir
[0mDEBUG   [U=1003,P=27606]   s_mkpath()                                Opps, could not create directory /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir: (2) No such file or directory
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      singularity_runtime_mount_cwd()           Creating current directory (privileged) within container: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir/my_user/Code/singularity
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir/my_user
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir/my_user/Code
[0mDEBUG   [U=0,P=27606]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/workdir/my_user/Code/singularity
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_mount_cwd()           Binding '/workdir/my_user/Code/singularity' to '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final//workdir/my_user/Code/singularity'
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files()               Running file components
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_files_passwd()        Called singularity_file_passwd_create()
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_files_passwd()        Checking configuration option: 'config passwd'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config passwd, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value config passwd='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config passwd, yes) = 1
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_passwd()        Checking for template passwd file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/passwd
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_passwd()        Creating template of /etc/passwd
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Called copy_file(/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/passwd, /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/passwd)
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Opening source file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/passwd
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Opening destination file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/passwd
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Calling fstat() on source file descriptor: 6
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Cloning permission string of source to dest
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Copying file data...
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Done copying data, closing file pointers
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Returning copy_file(/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/passwd, /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/passwd) = 0
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_passwd()        Creating template passwd file and appending user data: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/passwd
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   container_file_bind()                     Called file_bind(/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/passwd, /etc/passwd()
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      container_file_bind()                     Binding file '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/passwd' to '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/passwd'
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   envar_set()                               Setting environment variable: 'HOME' = '/home/my_user'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_files_group()         Called singularity_file_group_create()
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_files_group()         Checking configuration option: 'config group'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config group, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value config group='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config group, yes) = 1
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Creating template of /etc/group for containment
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Called copy_file(/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/group, /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/group)
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Opening source file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/group
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Opening destination file: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/group
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Calling fstat() on source file descriptor: 6
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Cloning permission string of source to dest
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Copying file data...
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Done copying data, closing file pointers
[0mDEBUG   [U=1003,P=27606]   copy_file()                               Returning copy_file(/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/group, /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/group) = 0
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Updating group file with user info
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_files_group()         Getting supplementary group info
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_files_group()         Skipping duplicate supplementary group
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Found supplementary group membership in: 3001
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Adding user's supplementary group ('docker') info to template group file
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Found supplementary group membership in: 3003
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Adding user's supplementary group ('pygmalion') info to template group file
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Found supplementary group membership in: 3004
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_files_group()         Adding user's supplementary group ('pygmalion-ext') info to template group file
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   container_file_bind()                     Called file_bind(/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/group, /etc/group()
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      container_file_bind()                     Binding file '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session/group' to '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/group'
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_files_resolvconf()    Checking configuration option
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config resolv_conf, yes)
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_value_impl()       Returning configuration value config resolv_conf='yes'
[0mDEBUG   [U=1003,P=27606]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config resolv_conf, yes) = 1
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   container_file_bind()                     Called file_bind(/etc/resolv.conf, /etc/resolv.conf()
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      container_file_bind()                     Binding file '/etc/resolv.conf' to '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final/etc/resolv.conf'
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/2.3/var/singularity/mnt/session'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CONTAINLIBS'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mVERBOSE [U=1003,P=27606]   singularity_runtime_enter()               Containing all rootfs components
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mVERBOSE [U=0,P=27606]      singularity_runtime_enter_chroot()        Entering container file system root: /gpfs/opt/tools/singularity/2.3/var/singularity/mnt/overlay/final
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop()                   Restoring supplementary groups
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop()                   Confirming we have correct UID/GID
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_enter_chroot()        Changing dir to '/' within the new root
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Cloning environment
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Counted 64 environment elements
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CLEANENV'
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Cleaning environment
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: MANPATH=:/opt/c3/man:/gpfs/opt/pbs/default/man:/opt/sgi/share/man
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: HOSTNAME=r1i7n6
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SHELL=/bin/bash
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: HISTSIZE=1000
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: GROIMP_ROOT=/Code/GroIMP-1.5/
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBNAME=singularity
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_ENVIRONMENT=PBS_BATCH
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: QTDIR=/usr/lib64/qt-3.3
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: QTINC=/usr/lib64/qt-3.3/include
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_WORKDIR=/workdir/my_user/Code/singularity
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: NCPUS=4
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_libexecdir=/gpfs/opt/tools/singularity/2.3/libexec
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_libexecdir
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_COMMAND=exec
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_COMMAND
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: QT_GRAPHICSSYSTEM_CHECKED=1
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: USER=my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_TASKNUM=1
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_MESSAGELEVEL=5
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_MESSAGELEVEL
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_HOME=/home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: WORKDIR=/workdir/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_ROOTFS=/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_ROOTFS
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_MOMPORT=15003
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_version=2.3-dist
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_version
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_QUEUE=haswellq
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_LOGNAME=my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: MAIL=/var/spool/mail/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_localstatedir=/gpfs/opt/tools/singularity/2.3/var
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_localstatedir
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_LANG=en_US.UTF-8
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBCOOKIE=000000004FCCD0100000000070866F41
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PWD=/workdir/my_user/Code/singularity
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITYENV_HISTFILE=
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITYENV_PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_NODENUM=0
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: LANG=en_US.UTF-8
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: MODULEPATH=/usr/share/Modules/modulefiles:/gpfs/opt/modulefiles/compilers:/gpfs/opt/modulefiles/libraries:/gpfs/opt/modulefiles/tools:/gpfs/opt/modulefiles/debuggers_and_profilers:/gpfs/opt/modulefiles/languages:/gpfs/opt/modulefiles/softwares
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_LOCALCACHEDIR=/workdir/my_user/Code/singularity
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_LOCALCACHEDIR
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: TZ=Europe/Paris
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBDIR=/home/my_user/pbs.211504.my_cluster.x8z
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: LOADEDMODULES=
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_SHELL=/bin/bash
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: C3_RSH=ssh -oConnectTimeout=10 -oForwardX11=no
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBID=211504.my_cluster
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: ENVIRONMENT=BATCH
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: HISTCONTROL=ignoredups
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: HOME=/home/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SHLVL=2
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: LANGUAGE=en_US.UTF-8
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_HOST=my_cluster.ib0.ice.my_institute.fr
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CONTENTS=/tmp/.singularity-layers.cjnQiYDr
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTENTS
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_sysconfdir=/gpfs/opt/tools/singularity/2.3/etc
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_sysconfdir
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: LOGNAME=my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: QTLIB=/usr/lib64/qt-3.3/lib
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_bindir=/gpfs/opt/tools/singularity/2.3/bin
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_bindir
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CLEANUPDIR=/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CLEANUPDIR
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_QUEUE=haswell_shmq
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: MODULESHOME=/usr/share/Modules
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_MAIL=/var/spool/mail/my_user
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: OMP_NUM_THREADS=4
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: LESSOPEN=||/usr/bin/lesspipe.sh %s
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CONTAINER=docker://centos
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTAINER
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_IMAGE=/workdir/my_user/Code/singularity/.singularity-runtime.UGmdSYv4/centos
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_IMAGE
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_SYSTEM=Linux
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_NODEFILE=/var/spool/PBS/aux/211504.my_cluster
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_PATH=/workdir/my_user/Code/apache-maven-3.3.9/bin:/gpfs/opt/tools/singularity/2.3/bin/:/home/my_user/bin:/opt/sgi/sbin:/opt/sgi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/c3/bin:/gpfs/opt/bin:/usr/lpp/mmfs/bin:/opt/pbs/default/bin:/opt/pbs/default/sbin:/sbin:/bin
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Transposing environment
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Converting envar 'SINGULARITYENV_HISTFILE' to 'HISTFILE' = '(null)'
[0mDEBUG   [U=1003,P=27606]   envar_set()                               Unsetting environment variable: HISTFILE
[0mDEBUG   [U=1003,P=27606]   singularity_runtime_environment()         Converting envar 'SINGULARITYENV_PATH' to 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin'
[0mDEBUG   [U=1003,P=27606]   envar_set()                               Setting environment variable: 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin'
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop_perm()              Called singularity_priv_drop_perm(void)
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop_perm()              Escalating permissison so we can properly drop permission
[0mDEBUG   [U=1003,P=27606]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
[0mDEBUG   [U=0,P=27606]      singularity_priv_escalate()               Clearing supplementary GIDs.
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop_perm()              Resetting supplementary groups
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop_perm()              Dropping to group ID '1200'
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop_perm()              Dropping real and effective privileges to GID = '1200'
[0mDEBUG   [U=0,P=27606]      singularity_priv_drop_perm()              Dropping real and effective privileges to UID = '1003'
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop_perm()              Confirming we have correct GID
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop_perm()              Confirming we have correct UID
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop_perm()              Setting NO_NEW_PRIVS to prevent future privilege escalations.
[0mDEBUG   [U=1003,P=27606]   singularity_priv_drop_perm()              Finished dropping privileges
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'CONTAIN'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'TARGET_PWD'
[0mDEBUG   [U=1003,P=27606]   main()                                    Attempting to chdir to CWD: /workdir/my_user/Code/singularity
[0mDEBUG   [U=1003,P=27606]   envar_set()                               Setting environment variable: 'SINGULARITY_CONTAINER' = 'centos'
[0mDEBUG   [U=1003,P=27606]   envar_set()                               Setting environment variable: 'SINGULARITY_NAME' = 'centos'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning NULL on 'SHELL'
[0mDEBUG   [U=1003,P=27606]   envar_set()                               Unsetting environment variable: SINGULARITY_SHELL
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'COMMAND' = 'exec'
[0mDEBUG   [U=1003,P=27606]   singularity_registry_get()                Returning value from registry: 'COMMAND' = 'exec'
[0m[34mLOG     [U=1003,P=27606]   main()                                    USER=my_user, IMAGE='centos', COMMAND='exec'
[0mDEBUG   [U=1003,P=27606]   action_exec()                             Checking for: /.singularity.d/actions/exec
[0mVERBOSE [U=1003,P=27606]   action_exec()                             Exec'ing /.singularity.d/actions/exec
[0mCentOS Linux release 7.3.1611 (Core) 
0
ls: cannot access .singularity_runtime*: No such file or directory
2

gmkurtzer commented 7 years ago

Do you have any errors showing up in syslog?

Also, I noticed that you are not using a local state directory. I would recommend compiling Singularity with --localstatedir=/var/ to ensure it is always on the local file system.

Greg

Quanteek commented 7 years ago

There is no syslog file in the /var/log of the node.

We recompiled singularity with the option you mentioned but it did not change the behaviour.

gmkurtzer commented 7 years ago

I just committed a change to the development branch that allows for debugging of the cleanup thread to STDERR when in --debug mode. Can you update to that version, and rerun the command, again with debugging enabled, and send the output please?

Thank you!

hra0031 commented 7 years ago

We are experiencing a similar problem. The cleanup process can't remove usr/share/zoneinfo/UTC and exit. (CentOS 6.7/6.9)

exit
VERBOSE [U=4573,P=21391]   main()                                    Cleaning directory: /tmp/.singularity-runtime.NpHtCEyE
DEBUG   [U=4573,P=21391]   s_rmdir()                                 Removing directory: /tmp/.singularity-runtime.NpHtCEyE
WARNING [U=4573,P=21391]   writable()                                Failed changing permission of file: /tmp/.singularity-runtime.NpHtCEyE/centos:latest/home/hra0031
WARNING [U=4573,P=21391]   unlink()                                  Failed removing file: /tmp/.singularity-runtime.NpHtCEyE/centos:latest/usr/share/zoneinfo/UTC
ERROR   [U=4573,P=21391]   main()                                    Could not remove directory /tmp/.singularity-runtime.NpHtCEyE: Device or resource busy
ABORT   [U=4573,P=21391]   main()                                    Retval = 255

In the output, I'm missing debug information from the original thread. I can see only debug output after daemonize. When I try to remove the file manually it is working without problem.

Quanteek commented 7 years ago

Here is the log (localstatevar has been disabled this time)

Enabling debugging
Ending argument loop
Singularity version: 2.3-development.g6755b2f
Exec'ing: /gpfs/opt/tools/singularity/dev/libexec/singularity/cli/exec.exec
Evaluating args: 'docker://centos cat /etc/redhat-release'
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos
VERBOSE2 SINGULARITY_METADATA_FOLDER found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /home/my_user/.singularity
VERBOSE2 SINGULARITY_ENVIRONMENT found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos/.singularity.d/environment
VERBOSE2 SINGULARITY_RUNSCRIPT found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos/singularity
VERBOSE2 SINGULARITY_TESTFILE found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos/.singularity.d/test
VERBOSE2 SINGULARITY_DEFFILE found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos/.singularity.d/Singularity
VERBOSE2 SINGULARITY_ENVBASE found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as /gpfs/workdir/my_user/Code/singularity
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layers.d5ursVW6
VERBOSE2 SINGULARITY_PYTHREADS found as 9
VERBOSE2 SINGULARITY_CONTAINER found as docker://centos
DEBUG Found uri docker://
DEBUG 
*** STARTING DOCKER IMPORT PYTHON  ****
DEBUG Docker layers and (env,labels,runscript) will be written to: /tmp/.singularity-layers.d5ursVW6
VERBOSE2 SINGULARITY_DOCKER_USERNAME not defined (None)
VERBOSE2 SINGULARITY_DOCKER_PASSWORD found
DEBUG Starting Docker IMPORT, includes environment, runscript, and metadata.
VERBOSE Docker image: centos
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: index.docker.io
VERBOSE Namespace: library
VERBOSE Repo Name: centos
VERBOSE Repo Tag: latest
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/library/centos/tags/list
DEBUG GET https://index.docker.io/v2/library/centos/tags/list
DEBUG Http Error with code 401
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:library/centos:pull
DEBUG Headers found: Content-Type,Authorization,Accept
VERBOSE3 Response on obtaining token is None.
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/centos/manifests/latest
DEBUG GET https://index.docker.io/v2/library/centos/manifests/latest
DEBUG Image manifest version 2.2 found.
DEBUG Adding digest sha256:d5e46245fe40c2d1ab72bfe328de28549b605b2587ab2fa8715f54e3e2de9c5d
VERBOSE Obtaining manifest: https://index.docker.io/v2/library/centos/manifests/latest
DEBUG GET https://index.docker.io/v2/library/centos/manifests/latest
DEBUG Using 9 workers for multiprocess.
VERBOSE3 Found Docker command (Entrypoint) None
VERBOSE3 Found Docker command (Cmd) /bin/bash
VERBOSE3 Adding Docker CMD as Singularity runscript...
DEBUG /bin/bash
VERBOSE3 Found Docker command (Env) PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
VERBOSE3 Found Docker container environment!
VERBOSE3 Adding Docker environment to metadata tar
DEBUG Found template for tarinfo
VERBOSE3 Found Docker command (Labels) {u'build-date': u'20170605', u'vendor': u'CentOS', u'name': u'CentOS Base Image', u'license': u'GPLv2'}
VERBOSE3 Found Docker container labels!
VERBOSE3 Adding Docker labels to metadata tar
DEBUG Found template for tarinfo
VERBOSE3 Adding Docker runscript to metadata tar
DEBUG Found template for tarinfo
DEBUG metadata file /gpfs/home/my_user/.singularity/metadata/sha256:6b8bbe197a20c88d065c265cf6f6f8b4e3695f104d1f47f01a1298b3566f27fe.tar.gz already exists, will over-write.
VERBOSE2 Tar file with Docker env and labels: /gpfs/home/my_user/.singularity/metadata/sha256:6b8bbe197a20c88d065c265cf6f6f8b4e3695f104d1f47f01a1298b3566f27fe.tar.gz
VERBOSE3 Writing Docker layers files to /tmp/.singularity-layers.d5ursVW6
VERBOSE2 Writing file /tmp/.singularity-layers.d5ursVW6 with mode w.
VERBOSE2 Writing file /tmp/.singularity-layers.d5ursVW6 with mode a.
DEBUG *** FINISHING DOCKER IMPORT PYTHON PORTION ****
Docker image path: index.docker.io/library/centos:latest
Cache folder set to /gpfs/home/my_user/.singularity/docker
Creating container runtime...
Importing: base Singularity environment
Exploding layer: sha256:d5e46245fe40c2d1ab72bfe328de28549b605b2587ab2fa8715f54e3e2de9c5d.tar.gz
Exploding layer: sha256:6b8bbe197a20c88d065c265cf6f6f8b4e3695f104d1f47f01a1298b3566f27fe.tar.gz
VERBOSE [U=0,P=12466]      message_init()                            Set messagelevel to: 5
VERBOSE [U=0,P=12466]      singularity_config_parse()                Initialize configuration file: /gpfs/opt/tools/singularity/dev/etc/singularity/singularity.conf
DEBUG   [U=0,P=12466]      singularity_config_parse()                Starting parse of configuration file /gpfs/opt/tools/singularity/dev/etc/singularity/singularity.conf
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key bind path = '/etc/localtime'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key enable overlay = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=0,P=12466]      singularity_config_parse()                Got config key sessiondir max size = '16'
DEBUG   [U=0,P=12466]      singularity_config_parse()                Finished parsing configuration file '/gpfs/opt/tools/singularity/dev/etc/singularity/singularity.conf'
VERBOSE [U=0,P=12466]      singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/gpfs/opt/tools/singularity/dev/libexec'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(libexecdir, /gpfs/opt/tools/singularity/dev/libexec) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'COMMAND' = 'exec'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(COMMAND, exec) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'ROOTFS' = '/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(ROOTFS, /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'VERSION' = '2.3-development.g6755b2f'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(version, 2.3-development.g6755b2f) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/gpfs/opt/tools/singularity/dev/var'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(localstatedir, /gpfs/opt/tools/singularity/dev/var) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'LOCALCACHEDIR' = '/workdir/my_user/Code/singularity'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(LOCALCACHEDIR, /workdir/my_user/Code/singularity) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'CONTENTS' = '/tmp/.singularity-layers.d5ursVW6'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(CONTENTS, /tmp/.singularity-layers.d5ursVW6) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/gpfs/opt/tools/singularity/dev/etc'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /gpfs/opt/tools/singularity/dev/etc) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'BINDIR' = '/gpfs/opt/tools/singularity/dev/bin'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(bindir, /gpfs/opt/tools/singularity/dev/bin) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(CLEANUPDIR, /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'CONTAINER' = 'docker://centos'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(CONTAINER, docker://centos) = 0
VERBOSE [U=0,P=12466]      singularity_registry_set()                Adding value to registry: 'IMAGE' = '/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos'
DEBUG   [U=0,P=12466]      singularity_registry_set()                Returning singularity_registry_set(IMAGE, /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos) = 0
DEBUG   [U=0,P=12466]      singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=0,P=12466]      singularity_registry_get()                Returning NULL on 'TARGET_UID'
DEBUG   [U=0,P=12466]      singularity_registry_get()                Returning NULL on 'TARGET_GID'
DEBUG   [U=0,P=12466]      singularity_priv_init()                   Initializing user info
DEBUG   [U=0,P=12466]      singularity_priv_init()                   Set the calling user's username to: my_user
DEBUG   [U=0,P=12466]      singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=0,P=12466]      singularity_priv_init()                   Obtaining home directory
VERBOSE [U=0,P=12466]      singularity_priv_init()                   Set home (via getpwuid()) to: /home/my_user
VERBOSE [U=0,P=12466]      singularity_suid_init()                   Running SUID program workflow
VERBOSE [U=0,P=12466]      singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=0,P=12466]      singularity_suid_init()                   Checking configuration file is properly owned by root
VERBOSE [U=0,P=12466]      singularity_suid_init()                   Checking if singularity.conf allows us to run as suid
DEBUG   [U=0,P=12466]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow setuid, yes)
DEBUG   [U=0,P=12466]      singularity_config_get_value_impl()       Returning configuration value allow setuid='yes'
DEBUG   [U=0,P=12466]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow setuid, yes) = 1
DEBUG   [U=0,P=12466]      singularity_registry_get()                Returning NULL on 'NOSUID'
VERBOSE [U=0,P=12466]      singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=0,P=12466]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=0,P=12466]      singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=0,P=12466]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
VERBOSE [U=0,P=12466]      singularity_priv_userns()                 Not virtualizing USER namespace: running as SUID
DEBUG   [U=0,P=12466]      singularity_priv_userns()                 Returning singularity_priv_init(void)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_ns()                  Calling: _singularity_runtime_ns_pid()
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow pid ns, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value allow pid ns='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow pid ns, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'UNSHARE_PID'
VERBOSE [U=1003,P=12466]   singularity_runtime_ns_pid()              Not virtualizing PID namespace on user request
DEBUG   [U=1003,P=12466]   singularity_runtime_ns()                  Calling: _singularity_runtime_ns_ipc()
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow ipc ns, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       No configuration entry found for 'allow ipc ns'; returning default value 'yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow ipc ns, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'UNSHARE_IPC'
VERBOSE [U=1003,P=12466]   singularity_runtime_ns_ipc()              Not virtualizing IPC namespace on user request
DEBUG   [U=1003,P=12466]   singularity_runtime_ns()                  Calling: _singularity_runtime_ns_mnt()
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount slave, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount slave='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount slave, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      singularity_runtime_ns_mnt()              Virtualizing FS namespace
DEBUG   [U=0,P=12466]      singularity_runtime_ns_mnt()              Virtualizing mount namespace
DEBUG   [U=0,P=12466]      singularity_runtime_ns_mnt()              Making mounts slave
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Setting sessiondir
VERBOSE [U=1003,P=12466]   singularity_sessiondir()                  Using session directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Checking for session directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Obtaining the default sessiondir size
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value sessiondir max size='16'
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Converted sessiondir size to: 16
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Creating the sessiondir size mount option length
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Got size length of: 9
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Creating the sessiondir size mount option string
DEBUG   [U=1003,P=12466]   singularity_sessiondir()                  Checking to make sure the string was allocated correctly
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      singularity_sessiondir()                  Mounting sessiondir tmpfs: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
VERBOSE [U=1003,P=12466]   singularity_registry_set()                Adding value to registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session'
DEBUG   [U=1003,P=12466]   singularity_registry_set()                Returning singularity_registry_set(SESSIONDIR, /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session) = 0
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'NOSESSIONCLEANUP'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'NOCLEANUP'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session'
DEBUG   [U=1003,P=12466]   singularity_cleanupd()                    Creating new cleanup trigger file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/cleanup_trigger
DEBUG   [U=1003,P=12466]   singularity_cleanupd()                    Opening cleanup trigger file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/cleanup_trigger
DEBUG   [U=1003,P=12466]   singularity_cleanupd()                    Gaining an exclusive flock on FD 4
VERBOSE [U=1003,P=23061]   singularity_cleanupd()                    Exec'ing cleanupd thread: /gpfs/opt/tools/singularity/dev/libexec/singularity/bin/cleanupd
DEBUG   [U=1003,P=23061]   envar_set()                               Setting environment variable: 'SINGULARITY_CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY'
DEBUG   [U=1003,P=23061]   envar_set()                               Setting environment variable: 'SINGULARITY_CLEANUPTRIGGER' = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/cleanup_trigger'
VERBOSE [U=1003,P=23061]   message_init()                            Set messagelevel to: 5
DEBUG   [U=1003,P=23061]   envar_path()                              Checking environment variable is valid path: 'SINGULARITY_CLEANUPDIR'
VERBOSE [U=1003,P=23061]   envar_get()                               Checking input from environment: 'SINGULARITY_CLEANUPDIR'
DEBUG   [U=1003,P=23061]   envar_get()                               Checking environment variable is defined: SINGULARITY_CLEANUPDIR
DEBUG   [U=1003,P=23061]   envar_get()                               Checking environment variable length (<= 4096): SINGULARITY_CLEANUPDIR
DEBUG   [U=1003,P=23061]   envar_get()                               Checking environment variable has allowed characters: SINGULARITY_CLEANUPDIR
VERBOSE [U=1003,P=23061]   envar_get()                               Obtained input from environment 'SINGULARITY_CLEANUPDIR' = '/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY'
DEBUG   [U=1003,P=23061]   envar_path()                              Checking environment variable is valid path: 'SINGULARITY_CLEANUPTRIGGER'
VERBOSE [U=1003,P=23061]   envar_get()                               Checking input from environment: 'SINGULARITY_CLEANUPTRIGGER'
DEBUG   [U=1003,P=23061]   envar_get()                               Checking environment variable is defined: SINGULARITY_CLEANUPTRIGGER
DEBUG   [U=1003,P=23061]   envar_get()                               Checking environment variable length (<= 4096): SINGULARITY_CLEANUPTRIGGER
DEBUG   [U=1003,P=23061]   envar_get()                               Checking environment variable has allowed characters: SINGULARITY_CLEANUPTRIGGER
VERBOSE [U=1003,P=23061]   envar_get()                               Obtained input from environment 'SINGULARITY_CLEANUPTRIGGER' = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/cleanup_trigger'
DEBUG   [U=1003,P=23061]   main()                                    Starting cleanup process
DEBUG   [U=1003,P=23061]   main()                                    Opening cleanup trigger file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/cleanup_trigger
DEBUG   [U=1003,P=23061]   main()                                    Checking to see if we need to daemonize
VERBOSE [U=1003,P=23061]   main()                                    Daemonizing cleandir cleanup process
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'IMAGE' = '/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'WRITABLE'
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       No configuration entry found for 'limit container owners'; returning default value 'NULL'
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       No configuration entry found for 'limit container paths'; returning default value 'NULL'
DEBUG   [U=1003,P=23062]   main()                                    Waiting for exclusive flock() on trigger file descriptor: 4
DEBUG   [U=1003,P=12466]   singularity_image_open()                  Opening file descriptor to image: /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos
VERBOSE [U=1003,P=12466]   singularity_image_check()                 Image is a directory, returning retval=1: /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value max loop devices='256'
DEBUG   [U=1003,P=12466]   singularity_image_bind()                  Entered singularity_image_bind()
DEBUG   [U=1003,P=12466]   singularity_image_bind()                  Converting max_loop_devs_string to int: '256'
DEBUG   [U=1003,P=12466]   singularity_image_bind()                  Converted max_loop_devs_string to int: '256' -> 256
DEBUG   [U=1003,P=12466]   singularity_image_bind()                  Checking if this image has been properly opened
DEBUG   [U=1003,P=12466]   singularity_image_bind()                  Checking if image is valid file
VERBOSE [U=1003,P=12466]   singularity_image_bind()                  Skipping image bind, container is not a file
VERBOSE [U=1003,P=12466]   singularity_runtime_rootfs()              Set container directory to: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Checking for container directory
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
DEBUG   [U=1003,P=12466]   chk_mode()                                Checking exact mode (40755) on: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
DEBUG   [U=1003,P=12466]   chk_mode()                                Found appropriate mode on file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
VERBOSE [U=1003,P=12466]   singularity_image_mount()                 Checking what kind of image we are mounting
DEBUG   [U=1003,P=12466]   singularity_image_mount_squashfs_check()  Image does not appear to be of type '.sqsh': /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos
DEBUG   [U=1003,P=12466]   singularity_image_mount_dir_check()       Checking if image is a directory
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      singularity_image_mount_dir_mount()       Mounting container directory /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos->/gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
DEBUG   [U=1003,P=12466]   action_ready()                            Checking if container is valid at: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/container
DEBUG   [U=1003,P=12466]   singularity_runtime_overlayfs()           Checking if overlayfs should be used
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(enable overlay, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value enable overlay='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(enable overlay, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'DISABLE_OVERLAYFS'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'WRITABLE'
DEBUG   [U=1003,P=12466]   singularity_runtime_overlayfs()           OverlayFS enabled by host build
DEBUG   [U=1003,P=12466]   singularity_runtime_overlayfs()           Setting up overlay mount options
DEBUG   [U=1003,P=12466]   singularity_runtime_overlayfs()           Checking for existance of overlay directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      singularity_runtime_overlayfs()           Mounting overlay tmpfs: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay
DEBUG   [U=0,P=12466]      singularity_runtime_overlayfs()           Creating upper overlay directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/upper
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/upper
DEBUG   [U=0,P=12466]      singularity_runtime_overlayfs()           Creating overlay work directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/work
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/work
DEBUG   [U=0,P=12466]      singularity_runtime_overlayfs()           Creating overlay_final directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
VERBOSE [U=0,P=12466]      singularity_runtime_overlayfs()           Mounting overlay with options: lowerdir=/gpfs/opt/tools/singularity/dev/var/singularity/mnt/container,upperdir=/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/upper,workdir=/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/work
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
VERBOSE [U=1003,P=12466]   singularity_registry_set()                Adding value to registry: 'OVERLAYFS_ENABLED' = '1'
DEBUG   [U=1003,P=12466]   singularity_registry_set()                Returning singularity_registry_set(OVERLAYFS_ENABLED, 1) = 0
VERBOSE [U=1003,P=12466]   singularity_runtime_overlayfs()           Updating the containerdir to: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Setting container_directory = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final'
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
VERBOSE [U=1003,P=12466]   singularity_runtime_mounts()              Running all mount components
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount hostfs, no)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount hostfs='no'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount hostfs, no) = 0
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_hostfs()        Not mounting host file systems per configuration
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_binds()         Checking configuration file for 'bind path'
VERBOSE [U=1003,P=12466]   singularity_runtime_mount_binds()         Found 'bind path' = /etc/localtime, /etc/localtime
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_binds()         Checking if bind point is already mounted: /etc/localtime
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   check_mounted()                           Opening /proc/mounts
DEBUG   [U=1003,P=12466]   check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_binds()         Binding '/etc/localtime' to '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final//etc/localtime'
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
VERBOSE [U=1003,P=12466]   singularity_runtime_mount_binds()         Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_binds()         Checking if bind point is already mounted: /etc/hosts
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   check_mounted()                           Opening /proc/mounts
DEBUG   [U=1003,P=12466]   check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_binds()         Binding '/etc/hosts' to '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final//etc/hosts'
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount proc'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount proc, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount proc='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount proc, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'PIDNS_ENABLED'
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_kernelfs()      Bind-mounting host /proc
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount sys'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount sys, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount sys='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount sys, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_kernelfs()      Mounting /sys
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_dev()           Checking configuration file for 'mount dev'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount dev, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount dev, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_dev()           Bind mounting /dev
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount home, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount home='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount home, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Checking that home directry is configured: /home/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Checking if SINGULARITY_CONTAIN is set
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Checking if home directories are being influenced by user
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Checking ownership of home directory source: /home/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Checking to make sure home directory destination is a full path: /home/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Checking if home directory is already mounted: /home/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   check_mounted()                           Opening /proc/mounts
DEBUG   [U=1003,P=12466]   check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Checking if overlay is enabled
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'OVERLAYFS_ENABLED' = '1'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Binding home directory direct (no staging)
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Setting home directory source to: '/home/my_user'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Setting home directory dest to: '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final' + '/home/my_user'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Creating home directry within container: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/home/my_user
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/home/my_user
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_home()          Created home directory within the container: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/home/my_user
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_home()          Mounting home directory source into container: /home/my_user -> /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/home/my_user
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   envar_set()                               Setting environment variable: 'HOME' = '/home/my_user'
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_userbinds()     Checking for environment variable 'SINGULARITY_BINDPATH'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'BINDPATH'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_userbinds()     No user bind mounts specified.
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount tmp, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value mount tmp='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount tmp, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   check_mounted()                           Opening /proc/mounts
DEBUG   [U=1003,P=12466]   check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_tmp()           Mounting directory: /tmp
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   check_mounted()                           Opening /proc/mounts
DEBUG   [U=1003,P=12466]   check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_mount_tmp()           Mounting directory: /var/tmp
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_scratch()       Getting SINGULARITY_SCRATCHDIR from environment
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'SCRATCHDIR'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_scratch()       Not mounting scratch directory: Not requested
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Checking to see if we should mount current working directory
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Getting current working directory
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Checking if current directory exists in container
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Container does not have the directory: /workdir/my_user/Code/singularity
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Checking for contain option
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Checking if CWD is already mounted: /workdir/my_user/Code/singularity
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   check_mounted()                           Opening /proc/mounts
DEBUG   [U=1003,P=12466]   check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Checking if cwd is in an operating system directory
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Checking if overlay is enabled
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'OVERLAYFS_ENABLED' = '1'
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Checking configuration file for 'user bind control'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(user bind control, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value user bind control='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(user bind control, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_runtime_mount_cwd()           Creating current directory within container: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir/my_user/Code/singularity
DEBUG   [U=1003,P=12466]   s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir
DEBUG   [U=1003,P=12466]   s_mkpath()                                Opps, could not create directory /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir: (2) No such file or directory
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      singularity_runtime_mount_cwd()           Creating current directory (privileged) within container: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir/my_user/Code/singularity
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir/my_user
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir/my_user/Code
DEBUG   [U=0,P=12466]      s_mkpath()                                Creating directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/workdir/my_user/Code/singularity
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
VERBOSE [U=1003,P=12466]   singularity_runtime_mount_cwd()           Binding '/workdir/my_user/Code/singularity' to '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final//workdir/my_user/Code/singularity'
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
VERBOSE [U=1003,P=12466]   singularity_runtime_files()               Running file components
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session'
DEBUG   [U=1003,P=12466]   singularity_runtime_files_passwd()        Called singularity_file_passwd_create()
DEBUG   [U=1003,P=12466]   singularity_runtime_files_passwd()        Checking configuration option: 'config passwd'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config passwd, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value config passwd='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config passwd, yes) = 1
VERBOSE [U=1003,P=12466]   singularity_runtime_files_passwd()        Checking for template passwd file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/passwd
VERBOSE [U=1003,P=12466]   singularity_runtime_files_passwd()        Creating template of /etc/passwd
DEBUG   [U=1003,P=12466]   copy_file()                               Called copy_file(/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/passwd, /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/passwd)
DEBUG   [U=1003,P=12466]   copy_file()                               Opening source file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/passwd
DEBUG   [U=1003,P=12466]   copy_file()                               Opening destination file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/passwd
DEBUG   [U=1003,P=12466]   copy_file()                               Calling fstat() on source file descriptor: 6
DEBUG   [U=1003,P=12466]   copy_file()                               Cloning permission string of source to dest
DEBUG   [U=1003,P=12466]   copy_file()                               Copying file data...
DEBUG   [U=1003,P=12466]   copy_file()                               Done copying data, closing file pointers
DEBUG   [U=1003,P=12466]   copy_file()                               Returning copy_file(/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/passwd, /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/passwd) = 0
VERBOSE [U=1003,P=12466]   singularity_runtime_files_passwd()        Creating template passwd file and appending user data: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/passwd
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   container_file_bind()                     Called file_bind(/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/passwd, /etc/passwd()
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      container_file_bind()                     Binding file '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/passwd' to '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/passwd'
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   envar_set()                               Setting environment variable: 'HOME' = '/home/my_user'
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session'
DEBUG   [U=1003,P=12466]   singularity_runtime_files_group()         Called singularity_file_group_create()
DEBUG   [U=1003,P=12466]   singularity_runtime_files_group()         Checking configuration option: 'config group'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config group, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value config group='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config group, yes) = 1
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Creating template of /etc/group for containment
DEBUG   [U=1003,P=12466]   copy_file()                               Called copy_file(/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/group, /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/group)
DEBUG   [U=1003,P=12466]   copy_file()                               Opening source file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/group
DEBUG   [U=1003,P=12466]   copy_file()                               Opening destination file: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/group
DEBUG   [U=1003,P=12466]   copy_file()                               Calling fstat() on source file descriptor: 6
DEBUG   [U=1003,P=12466]   copy_file()                               Cloning permission string of source to dest
DEBUG   [U=1003,P=12466]   copy_file()                               Copying file data...
DEBUG   [U=1003,P=12466]   copy_file()                               Done copying data, closing file pointers
DEBUG   [U=1003,P=12466]   copy_file()                               Returning copy_file(/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/group, /gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/group) = 0
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Updating group file with user info
DEBUG   [U=1003,P=12466]   singularity_runtime_files_group()         Getting supplementary group info
DEBUG   [U=1003,P=12466]   singularity_runtime_files_group()         Skipping duplicate supplementary group
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Found supplementary group membership in: 3001
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Adding user's supplementary group ('docker') info to template group file
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Found supplementary group membership in: 3003
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Adding user's supplementary group ('pygmalion') info to template group file
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Found supplementary group membership in: 3004
VERBOSE [U=1003,P=12466]   singularity_runtime_files_group()         Adding user's supplementary group ('pygmalion-ext') info to template group file
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   container_file_bind()                     Called file_bind(/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/group, /etc/group()
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      container_file_bind()                     Binding file '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session/group' to '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/group'
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_files_resolvconf()    Checking configuration option
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config resolv_conf, yes)
DEBUG   [U=1003,P=12466]   singularity_config_get_value_impl()       Returning configuration value config resolv_conf='yes'
DEBUG   [U=1003,P=12466]   singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config resolv_conf, yes) = 1
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   container_file_bind()                     Called file_bind(/etc/resolv.conf, /etc/resolv.conf()
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      container_file_bind()                     Binding file '/etc/resolv.conf' to '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final/etc/resolv.conf'
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/gpfs/opt/tools/singularity/dev/var/singularity/mnt/session'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CONTAINLIBS'
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
VERBOSE [U=1003,P=12466]   singularity_runtime_enter()               Containing all rootfs components
DEBUG   [U=1003,P=12466]   singularity_runtime_rootfs()              Returning container_directory: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=12466]      singularity_runtime_enter_chroot()        Entering container file system root: /gpfs/opt/tools/singularity/dev/var/singularity/mnt/overlay/final
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Dropping privileges to UID=1003, GID=1200 (4 supplementary GIDs)
DEBUG   [U=0,P=12466]      singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=1003,P=12466]   singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=1003,P=12466]   singularity_runtime_enter_chroot()        Changing dir to '/' within the new root
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Cloning environment
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Counted 64 environment elements
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CLEANENV'
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Cleaning environment
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: MANPATH=:/opt/c3/man:/gpfs/opt/pbs/default/man:/opt/sgi/share/man
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: HOSTNAME=r1i7n7
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SHELL=/bin/bash
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: HISTSIZE=1000
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: GROIMP_ROOT=/Code/GroIMP-1.5/
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBNAME=singularity
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_ENVIRONMENT=PBS_BATCH
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: QTDIR=/usr/lib64/qt-3.3
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: QTINC=/usr/lib64/qt-3.3/include
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_WORKDIR=/workdir/my_user/Code/singularity
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: NCPUS=4
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_libexecdir=/gpfs/opt/tools/singularity/dev/libexec
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_libexecdir
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_COMMAND=exec
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_COMMAND
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: QT_GRAPHICSSYSTEM_CHECKED=1
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: USER=my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_TASKNUM=1
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_MESSAGELEVEL=5
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_MESSAGELEVEL
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_HOME=/home/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: WORKDIR=/workdir/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_ROOTFS=/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_ROOTFS
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_MOMPORT=15003
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_version=2.3-development.g6755b2f
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_version
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_QUEUE=haswellq
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_LOGNAME=my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: MAIL=/var/spool/mail/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_localstatedir=/gpfs/opt/tools/singularity/dev/var
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_localstatedir
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_LANG=en_US.UTF-8
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBCOOKIE=000000007BD638CB0000000036272EF5
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PWD=/workdir/my_user/Code/singularity
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITYENV_HISTFILE=
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITYENV_PATH=/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_NODENUM=0
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: LANG=en_US.UTF-8
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: MODULEPATH=/usr/share/Modules/modulefiles:/gpfs/opt/modulefiles/compilers:/gpfs/opt/modulefiles/libraries:/gpfs/opt/modulefiles/tools:/gpfs/opt/modulefiles/debuggers_and_profilers:/gpfs/opt/modulefiles/languages:/gpfs/opt/modulefiles/softwares
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_LOCALCACHEDIR=/workdir/my_user/Code/singularity
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_LOCALCACHEDIR
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: TZ=Europe/Paris
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBDIR=/home/my_user/pbs.216658.my_cluster.x8z
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: LOADEDMODULES=
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_SHELL=/bin/bash
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: C3_RSH=ssh -oConnectTimeout=10 -oForwardX11=no
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_JOBID=216658.my_cluster
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: ENVIRONMENT=BATCH
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: HISTCONTROL=ignoredups
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: HOME=/home/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SHLVL=2
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: LANGUAGE=en_US.UTF-8
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_HOST=my_cluster.ib0.ice.my_institution.fr
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CONTENTS=/tmp/.singularity-layers.d5ursVW6
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTENTS
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_sysconfdir=/gpfs/opt/tools/singularity/dev/etc
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_sysconfdir
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: LOGNAME=my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: QTLIB=/usr/lib64/qt-3.3/lib
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_bindir=/gpfs/opt/tools/singularity/dev/bin
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_bindir
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CLEANUPDIR=/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CLEANUPDIR
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_QUEUE=haswell_shmq
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: MODULESHOME=/usr/share/Modules
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_MAIL=/var/spool/mail/my_user
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: OMP_NUM_THREADS=4
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: LESSOPEN=||/usr/bin/lesspipe.sh %s
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CONTAINER=docker://centos
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTAINER
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_IMAGE=/workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY/centos
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_IMAGE
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_SYSTEM=Linux
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_NODEFILE=/var/spool/PBS/aux/216658.my_cluster
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Evaluating envar to clean: PBS_O_PATH=/workdir/my_user/Code/apache-maven-3.3.9/bin:/gpfs/opt/tools/singularity/dev/bin/:/home/my_user/bin:/opt/sgi/sbin:/opt/sgi/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/c3/bin:/gpfs/opt/bin:/usr/lpp/mmfs/bin:/opt/pbs/default/bin:/opt/pbs/default/sbin:/sbin:/bin
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Transposing environment
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Converting envar 'SINGULARITYENV_HISTFILE' to 'HISTFILE' = '(null)'
DEBUG   [U=1003,P=12466]   envar_set()                               Unsetting environment variable: HISTFILE
DEBUG   [U=1003,P=12466]   singularity_runtime_environment()         Converting envar 'SINGULARITYENV_PATH' to 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin'
DEBUG   [U=1003,P=12466]   envar_set()                               Setting environment variable: 'PATH' = '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin'
DEBUG   [U=1003,P=12466]   singularity_priv_drop_perm()              Called singularity_priv_drop_perm(void)
DEBUG   [U=1003,P=12466]   singularity_priv_drop_perm()              Escalating permissison so we can properly drop permission
DEBUG   [U=1003,P=12466]   singularity_priv_escalate()               Temporarily escalating privileges (U=1003)
DEBUG   [U=0,P=12466]      singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=12466]      singularity_priv_drop_perm()              Resetting supplementary groups
DEBUG   [U=0,P=12466]      singularity_priv_drop_perm()              Dropping to group ID '1200'
DEBUG   [U=0,P=12466]      singularity_priv_drop_perm()              Dropping real and effective privileges to GID = '1200'
DEBUG   [U=0,P=12466]      singularity_priv_drop_perm()              Dropping real and effective privileges to UID = '1003'
DEBUG   [U=1003,P=12466]   singularity_priv_drop_perm()              Confirming we have correct GID
DEBUG   [U=1003,P=12466]   singularity_priv_drop_perm()              Confirming we have correct UID
DEBUG   [U=1003,P=12466]   singularity_priv_drop_perm()              Setting NO_NEW_PRIVS to prevent future privilege escalations.
DEBUG   [U=1003,P=12466]   singularity_priv_drop_perm()              Finished dropping privileges
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'TARGET_PWD'
DEBUG   [U=1003,P=12466]   main()                                    Attempting to chdir to CWD: /workdir/my_user/Code/singularity
DEBUG   [U=1003,P=12466]   envar_set()                               Setting environment variable: 'SINGULARITY_CONTAINER' = 'centos'
DEBUG   [U=1003,P=12466]   envar_set()                               Setting environment variable: 'SINGULARITY_NAME' = 'centos'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning NULL on 'SHELL'
DEBUG   [U=1003,P=12466]   envar_set()                               Unsetting environment variable: SINGULARITY_SHELL
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'COMMAND' = 'exec'
DEBUG   [U=1003,P=12466]   singularity_registry_get()                Returning value from registry: 'COMMAND' = 'exec'
LOG     [U=1003,P=12466]   main()                                    USER=my_user, IMAGE='centos', COMMAND='exec'
DEBUG   [U=1003,P=12466]   action_exec()                             Checking for: /.singularity.d/actions/exec
VERBOSE [U=1003,P=12466]   action_exec()                             Exec'ing /.singularity.d/actions/exec
CentOS Linux release 7.3.1611 (Core) 
VERBOSE [U=1003,P=23062]   main()                                    Cleaning directory: /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY
DEBUG   [U=1003,P=23062]   s_rmdir()                                 Removing directory: /workdir/my_user/Code/singularity/.singularity-runtime.SyFX3JlY
0
ls: cannot access .singularity_runtime*: No such file or directory
2

Quanteek commented 7 years ago

My guess is that it is coming from this line : https://github.com/singularityware/singularity/blob/adf5259ac94d7b213c649f073dcda32459b0c8b7/src/util/file.c#L350

return(nftw(dir, _unlink, 32, FTW_DEPTH|FTW_MOUNT|FTW_PHYS));

Maybe the value of 32 for the typeflag is not good or maybe FTW_MOUNT is not good also since my workdir is not mount has the samelocation of the singularity bin or the location of the computation node.

Quanteek commented 7 years ago

For information I have tested with this script

#PBS -S /bin/bash
#PBS -N singularity
#PBS -j oe
#PBS -o logs/
#PBS -l walltime=12:00:00
#PBS -l select=1:ncpus=4:mem=20gb
#PBS -q haswellq

cd $PBS_O_WORKDIR

mkdir $PBS_JOBID

cd $PBS_JOBID

SINGULARITY_LOCALCACHEDIR=`pwd` singularity --debug exec docker://centos cat /etc/redhat-release; echo $? ; ls -d .singularity_runtime* ; echo $? ;

ls -al .singularity-runtime.*/centos/usr

cd ..

rm -rf $PBS_JOBID

and I have got this error (especially the end with the Warnings) I don't know if it is normal (see attachment) log2.txt

gmkurtzer commented 7 years ago

I wonder why it is failing changing permissions on those files. Can you show me an ls -l on some of those files, and their parent directories?

ls -l /workdir/my_user/Code/singularity/216989.my_cluster/.singularity-runtime.K0bc9ODB/centos/var/lib/rpm/Providename
ls -ld /workdir/my_user/Code/singularity/216989.my_cluster/.singularity-runtime.K0bc9ODB/centos/var/lib/rpm

Thanks!

Quanteek commented 7 years ago

Hey @gmkurtzer

I have this output for these files:

drwx------ 3 my_user mics 4096 Jun 20 14:31 .singularity-runtime.jmhW3y8Z
-rw-r--r-- 1 my_user mics 1572864 Jun  5 05:09 .singularity-runtime.jmhW3y8Z/centos/var/lib/rpm/Providename
drwxr-xr-x 2 my_user mics 4096 Jun 20 14:32 .singularity-runtime.jmhW3y8Z/centos/var/lib/rpm

I also noticed a strange behavior when I launch several times the same "singularity exec" like:

#PBS -S /bin/bash
#PBS -j oe
#PBS -o logs/
#PBS -l walltime=12:00:00
#PBS -l select=1:ncpus=4:mem=20gb
#PBS -q haswellq

cd $PBS_O_WORKDIR

mkdir $PBS_JOBID

cp p.txt $PBS_JOBID/

pushd $PBS_JOBID

export SINGULARITY_LOCALCACHEDIR=`pwd`

#Nothing here
ls -al .*

#First try
time singularity --debug exec docker://centos cat /etc/redhat-release
ls -al .
ls -l .singularity-runtime.*/centos/var/lib/rpm/Providename
ls -ld .singularity-runtime.*/centos/var/lib/rpm

#Second try
time singularity --debug exec docker://centos cat /etc/redhat-release
ls -al .
ls -l .singularity-runtime.*/centos/var/lib/rpm/Providename
ls -ld .singularity-runtime.*/centos/var/lib/rpm

#Third try
time singularity --debug exec docker://centos cat /etc/redhat-release
ls -al .
ls -l .singularity-runtime.*/centos/var/lib/rpm/Providename
ls -ld .singularity-runtime.*/centos/var/lib/rpm

ls -al .*

cp y.txt ..

popd

#rm -rf $PBS_JOBID

It will create several singularity-runtime folder but they will be deleted at a moment when they do a new singularity exec hence the last one is not deleted.

Maybe there is a threaded process that is not able to finish before the end of the qsub ? (but I tried a sleep at the end and it did not change anything)

I put the output file of the previous pbs script in attachment. 299625.fusion.OU.txt

vsoch commented 7 years ago

Hey @Quanteek it's been a long time since we discussed this issue! Could you try with singularity 2.4 (now master branch) or development branch, and see if you still have this issue? Take a look at build instead of bootstrap too:

sudo singularity build ubuntu.simg docker://ubuntu

novosirj commented 6 years ago

I'm experiencing this with singularity exec anyway on 2.4.2 (and 2.4-dist with PR1106 applied. Job has run, everything has exited, and now:

[root@cuda001 tmp]# ls -la 
total 4
...
-rw-r--r--  1 novosirj novosirj   0 Dec 11 18:40 .singularity-cleanuptrigger.0374OkU6
drwx------  3 novosirj novosirj  60 Dec 11 18:40 .singularity-runtime.rYKEwMRT

hra0031 commented 6 years ago

With version 2.4.2 I get only warning:

WARNING [U=4573,P=26096]   writable()                                Failed changing permission of file: /tmp/.singularity-runtime.kddd5Mhz/jdeathe/centos-ssh/home/hra0031

But cleanup process finished OK.

jscook2345 commented 6 years ago

@Quanteek @novosirj Are you still having issues with this? It appears @hra0031 was able to get things working with version 2.4.2

novosirj commented 6 years ago

Indeed, I still have the problem. I believe I've tested as far forward as 2.6.0.

The problem does not occur, so far as I know, outside of our job scheduler. The issue appears to be that Singularity terminates, the scheduler closes the the cgroup that the job was running in, and the cleanup process gets killed in the process. I'm not sure what to do about that, but it would seem to be a problem with the way Singularity cleans up, not the scheduler.

jscook2345 commented 6 years ago

@novosirj Which scheduler are you using?

dtrudg commented 4 years ago

This is not believed to be an issue under current Singularity 3.5. If you still have the issue under Singularity 3.5 please let us know. Thanks.

novosirj commented 4 years ago

By councidence, we upgraded to 3.5.2 just the other day and I tested it and found the same problem exists. I too was hoping 3.5 would change it.

I’ll gather the specifics. I had mentally marked this as something that was not going to be fixed and was going to require an epilogue, which isn’t ideal.

On Feb 5, 2020, at 10:20, David Trudgian notifications@github.com wrote:

This is not believed to be an issue under current Singularity 3.5. If you still have the issue under Singularity 3.5 please let us know. Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/sylabs/singularity/issues/731?email_source=notifications&email_token=AFDASVYZP27ELXCPMNWUS4TRBLKKFA5CNFSM4DOLYNAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK3ZVKA#issuecomment-582458024, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AFDASV7MRJEQGC2AQQQ64NLRBLKKFANCNFSM4DOLYNAA.

dtrudg commented 4 years ago

Do you have any specifics available for the scheduler etc.? If the scheduler allows enough time between issuing a SIGTERM and then either a SIGKILL or cgroup action that hard kills Singularity then it should clean up AFAIK.

dtrudg commented 4 years ago

Closing again as no follow-up since March.

At this point I'd ask a new issue is opened for 3.6.x if it still occurs. This issue was opened against old 2.x versions and the debug content etc here is not likely very relevant for the problem under the latest versions.

apptainer / singularity