search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

Fix permissions docker #538

Closed hra0031 closed 7 years ago

hra0031 commented 7 years ago

After untaring docker image, some files and folders are not writable. This is a problem in modified images, which contains more than 2 tar archives. The third archive then fails to extract due to permissions.

vsoch commented 7 years ago

What branch is this using? And is the order of extraction correct (we had an issue with order in the past)

hra0031 commented 7 years ago

We are using Develop branch. Maybe the order isn't correct because the added archive is extracted as first. But I think this is not the main problem because when I look at clean docker image after extracting, the permissions are as follows:

hra0031@login3:/tmp/.singularity-runtime.MIPgK4Od/container/centos/usr$ ll
total 64
dr-xr-x---  2 hra0031 hra0031 12288 Mar 15 21:00 bin
drwxr-x---  2 hra0031 hra0031  4096 Nov  5 16:38 etc
drwxr-x---  2 hra0031 hra0031  4096 Nov  5 16:38 games
drwxr-x---  3 hra0031 hra0031  4096 Mar 15 20:59 include
dr-xr-x--- 20 hra0031 hra0031  4096 Mar 15 21:00 lib
dr-xr-x--- 26 hra0031 hra0031 16384 Mar 15 21:00 lib64
drwxr-x---  8 hra0031 hra0031  4096 Mar 15 21:00 libexec
drwxr-x--- 12 hra0031 hra0031  4096 Mar 15 20:58 local
dr-xr-x---  2 hra0031 hra0031  4096 Mar 15 21:00 sbin
drwxr-x--- 52 hra0031 hra0031  4096 Mar 15 21:00 share
drwxr-x---  4 hra0031 hra0031  4096 Mar 15 20:58 src
lrwxrwxrwx  1 hra0031 hra0031    10 Mar 27 12:03 tmp -> ../var/tmp

On some folders are the write permissions missing. So the next archive can't be extracted here.

vsoch commented 7 years ago

@gmkurtzer this is the issue I was talking about last night - when you are up let's discuss how to fix. Either we can handle it in the C, or revert back to python doing it. @hra0031 the good news is that I think we understand this issue and know how to fix it! Just been actively breaking things, lol. Stay tuned :)_

gmkurtzer commented 7 years ago

Gotcha... I have an idea on how to fix it. Is there an easy reproducer I can use to test?

Thanks!

hra0031 commented 7 years ago

We make new simple test like this:

$cat Dockerfile
FROM centos:7
RUN yum install -y libXext.x86_64 libXt.x86_64 libXmu.x86_64
$docker build -t docker-test
$docker tag docker-test storage.it4i.cz/hra0031/docker-test
$docker push storage.it4i.cz/hra0031/docker-test

In docker all works OK. The problem is that the archive is split into multiple tar archives. Both archives contain same folders. So when the first archive is extracted, rights for the folders are set. When is the second archive trying to extract remaining files to the same folders as the first, the extraction fails due to access rights.

truatpasteurdotfr commented 7 years ago

somehow related to https://github.com/CentOS/sig-cloud-instance-images/issues/67 ? the perms of /usr/lib and others being dr-xr-xr-x by "design" on CentOS-6/7 and that a regular user needs to override the perms before writting inside (while docker|root just do it without asking permissions) ?

vsoch commented 7 years ago

Can you try with the (now) development branch? We had the wrong extraction of layers (order) and just reversed it.

hra0031 commented 7 years ago

I've try it right now with develop version and public repository:

hra0031@login4:~$ singularity --debug shell docker://blacklabelops/centos                                                                                                                                                        [6124/28968]
enabling debugging
ending argument loop
Execing: /home/hra0031/.local/easybuild/software/Singularity/2.2.99-GCC-6.3.0-2.27-develop2/libexec/singularity/cli/shell.exec docker://blacklabelops/centos
Importing: base Singularity environment
DEBUG:python:SINGULARITY_COMMAND_ASIS found as False
DEBUG:python:SINGULARITY_ROOTFS found as /tmp/.singularity-runtime.1dgMnlAV/container/blacklabelops/centos
DEBUG:python:SINGULARITY_METADATA_FOLDER found as /tmp/.singularity-runtime.1dgMnlAV/container/blacklabelops/centos/.singularity.d
DEBUG:python:SINGULARITY_DISABLE_CACHE found as False
DEBUG:python:SINGULARITY_CACHEDIR found as /home/hra0031/.singularity
DEBUG:python:SINGULARITY_ENVBASE found as /tmp/.singularity-runtime.1dgMnlAV/container/blacklabelops/centos/.singularity.d/env
DEBUG:python:SINGULARITY_LABELFILE found as /tmp/.singularity-runtime.1dgMnlAV/container/blacklabelops/centos/.singularity.d/labels.json
DEBUG:python:SINGULARITY_INCLUDECMD found as False
DEBUG:python:SINGULARITY_PULLFOLDER found as /home/hra0031
DEBUG:python:SINGULARITY_CONTENTS found as /tmp/.singularity-layers.FVVHYWmq
DEBUG:python:SINGULARITY_CONTAINER found as docker://blacklabelops/centos
DEBUG:python:Found uri docker://
DEBUG:python:Found uri docker://
INFO:python:
*** STARTING DOCKER IMPORT PYTHON  ****
INFO:python:Docker layers and (env,labels,runscript) will be written to: /tmp/.singularity-layers.FVVHYWmq
DEBUG:python:SINGULARITY_DOCKER_USERNAME not defined (None)
DEBUG:python:SINGULARITY_DOCKER_PASSWORD found
DEBUG:python:Starting Docker IMPORT, includes environment, runscript, and metadata.
INFO:python:Docker image: blacklabelops/centos
INFO:python:Specified Docker ENTRYPOINT as %runscript.
INFO:python:Registry: index.docker.io
INFO:python:Namespace: blacklabelops
INFO:python:Repo Name: centos
INFO:python:Repo Tag: latest
INFO:python:Version: None
DEBUG:python:Headers found: Content-Type,Accept
INFO:python:Obtaining tags: https://index.docker.io/v2/blacklabelops/centos/tags/list
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/tags/list
DEBUG:python:GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:blacklabelops/centos:pull
DEBUG:python:Headers found: Authorization,Content-Type,Accept
DEBUG:python:GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:blacklabelops/centos:pull
DEBUG:python:Headers found: Authorization,Content-Type,Accept
INFO:python:Docker image path: index.docker.io/blacklabelops/centos:latest
INFO:python:Obtaining manifest: https://index.docker.io/v2/blacklabelops/centos/manifests/latest
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/manifests/latest
INFO:python:Image manifest version 2.2 found.
INFO:python:Adding digest sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
INFO:python:Adding digest sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
INFO:python:Obtaining manifest: https://index.docker.io/v2/blacklabelops/centos/manifests/latest
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/manifests/latest
Cache folder set to /home/hra0031/.singularity/docker
INFO:python:Downloading layers from https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
Downloading layer sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
INFO:python:Downloading layers from https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
Downloading layer sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
INFO:python:Found Docker command (Entrypoint) None
INFO:python:Found Docker command (Cmd) /bin/bash
DEBUG:python:Adding Docker CMD as Singularity runscript...
DEBUG:python:/bin/bash
INFO:python:Found Docker command (Env) PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG:python:Found Docker container environment!
DEBUG:python:Adding Docker environment to metadata tar
INFO:python:Found Docker command (Labels) {'com.blacklabelops.maintainer.email': 'sbl@blacklabelops.com', 'maintainer': 'Steffen Bleul <sbl@blacklabelops.com>', 'com.blacklabelops.support': 'http://support.blacklabelops.com/', 'com.black
labelops.image.osversion': '7.2.1511', 'vendor': 'CentOS', 'name': 'CentOS Base Image', 'build-date': '20161214', 'com.blacklabelops.image.os': 'centos', 'com.blacklabelops.maintainer.name': 'Steffen Bleul', 'com.blacklabelops.image.buil
ddate.centos': '28/03/2017-01:05+0200', 'com.blacklabelops.image.name.centos': 'centos-base-image', 'license': 'GPLv2'}
DEBUG:python:Found Docker container labels!
DEBUG:python:Adding Docker labels to metadata tar
DEBUG:python:Adding Docker runscript to metadata tar
DEBUG:python:Tar file with Docker env and labels: /home/hra0031/.singularity/docker/sha256:3ddc4df119adfe8d1889183c268fb1289481da8b1a62731390925d09b8851d7a.tar.gz
DEBUG:python:Writing Docker layers files to /tmp/.singularity-layers.FVVHYWmq
INFO:python:Writing file /tmp/.singularity-layers.FVVHYWmq with mode w.
INFO:python:Writing file /tmp/.singularity-layers.FVVHYWmq with mode a.
INFO:python:*** FINISHING DOCKER IMPORT PYTHON PORTION ****
Exploding layer: sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7.tar.gz
Exploding layer: sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729.tar.gz
tar: usr/bin/[: Cannot open: Permission denied
tar: usr/bin/addr2line: Cannot open: Permission denied
tar: usr/bin/alias: Cannot open: Permission denied
tar: usr/bin/ar: Cannot open: Permission denied
tar: usr/bin/arch: Cannot open: Permission denied
tar: usr/bin/as: Cannot open: Permission denied
tar: usr/bin/awk: Cannot create symlink to gawk: Permission denied
tar: usr/bin/base64: Cannot open: Permission denied
tar: usr/bin/basename: Cannot open: Permission denied
...

@truatpasteurdotfr I think this could be right, because I've try the Ubuntu image right now and it's working OK

vsoch commented 7 years ago

We used to find files with missing permissions and fix them - @gmkurtzer can we do this equivalently with your extraction? If not we can go back to python. What are the cons of having the python portion extract the layers that would make it not suitable for a non sudo import?

vsoch commented 7 years ago

just tested this with the old version - @truatpasteurdotfr +1 I think this is a special case, because the same error is present with our previous version!

vsoch commented 7 years ago

ok, and just confirmed that it works with sudo and --writable

sudo singularity --debug shell --writable docker://blacklabelops/centos
[sudo] password for vanessa: 
enabling debugging
ending argument loop
Exec'ing: /usr/local/libexec/singularity/cli/shell.exec --writableImporting: base Singularity environment
DEBUG:python:SINGULARITY_COMMAND_ASIS found as False
DEBUG:python:SINGULARITY_ROOTFS found as /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos
DEBUG:python:SINGULARITY_METADATA_FOLDER found as /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos/.singularity.d
DEBUG:python:SINGULARITY_DISABLE_CACHE found as False
DEBUG:python:SINGULARITY_CACHEDIR found as /root/.singularity
DEBUG:python:SINGULARITY_ENVBASE found as /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos/.singularity.d/env
DEBUG:python:SINGULARITY_LABELFILE found as /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos/.singularity.d/labels.json
DEBUG:python:SINGULARITY_INCLUDECMD found as False
DEBUG:python:SINGULARITY_PULLFOLDER found as /home/vanessa
DEBUG:python:SINGULARITY_CONTENTS found as /tmp/.singularity-layers.c1DXDRgy
DEBUG:python:SINGULARITY_CONTAINER found as docker://blacklabelops/centos
DEBUG:python:Found uri docker://
DEBUG:python:Found uri docker://
INFO:python:
*** STARTING DOCKER IMPORT PYTHON  ****
INFO:python:Docker layers and (env,labels,runscript) will be written to: /tmp/.singularity-layers.c1DXDRgy
DEBUG:python:SINGULARITY_DOCKER_USERNAME not defined (None)
DEBUG:python:SINGULARITY_DOCKER_PASSWORD found
DEBUG:python:Starting Docker IMPORT, includes environment, runscript, and metadata.
INFO:python:Docker image: blacklabelops/centos
INFO:python:Specified Docker ENTRYPOINT as %runscript.
INFO:python:Registry: index.docker.io
INFO:python:Namespace: blacklabelops
INFO:python:Repo Name: centos
INFO:python:Repo Tag: latest
INFO:python:Version: None
DEBUG:python:Headers found: Content-Type,Accept
INFO:python:Obtaining tags: https://index.docker.io/v2/blacklabelops/centos/tags/list
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/tags/list
DEBUG:python:GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:blacklabelops/centos:pull
DEBUG:python:Headers found: Content-Type,Authorization,Accept
DEBUG:python:GET https://auth.docker.io/token?service=registry.docker.io&scope=repository:blacklabelops/centos:pull
DEBUG:python:Headers found: Content-Type,Authorization,Accept
INFO:python:Docker image path: index.docker.io/blacklabelops/centos:latest
INFO:python:Obtaining manifest: https://index.docker.io/v2/blacklabelops/centos/manifests/latest
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/manifests/latest
INFO:python:Image manifest version 2.2 found.
INFO:python:Adding digest sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
INFO:python:Adding digest sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
INFO:python:Obtaining manifest: https://index.docker.io/v2/blacklabelops/centos/manifests/latest
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/manifests/latest
Cache folder set to /root/.singularity/docker
INFO:python:Downloading layers from https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
Downloading layer sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729
cd INFO:python:Downloading layers from https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
Downloading layer sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
DEBUG:python:GET https://index.docker.io/v2/blacklabelops/centos/blobs/sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7
INFO:python:Found Docker command (Entrypoint) None
INFO:python:Found Docker command (Cmd) /bin/bash
DEBUG:python:Adding Docker CMD as Singularity runscript...
DEBUG:python:/bin/bash
INFO:python:Found Docker command (Env) PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
DEBUG:python:Found Docker container environment!
DEBUG:python:Adding Docker environment to metadata tar
INFO:python:Found Docker command (Labels) {u'com.blacklabelops.image.os': u'centos', u'maintainer': u'Steffen Bleul <sbl@blacklabelops.com>', u'com.blacklabelops.image.osversion': u'7.2.1511', u'license': u'GPLv2', u'com.blacklabelops.image.name.centos': u'centos-base-image', u'com.blacklabelops.image.builddate.centos': u'28/03/2017-01:05+0200', u'com.blacklabelops.maintainer.name': u'Steffen Bleul', u'com.blacklabelops.maintainer.email': u'sbl@blacklabelops.com', u'vendor': u'CentOS', u'build-date': u'20161214', u'com.blacklabelops.support': u'http://support.blacklabelops.com/', u'name': u'CentOS Base Image'}
DEBUG:python:Found Docker container labels!
DEBUG:python:Adding Docker labels to metadata tar
DEBUG:python:Adding Docker runscript to metadata tar
DEBUG:python:Tar file with Docker env and labels: /root/.singularity/docker/sha256:cebfcb455b0f307f2cc86b3c54677c4a78279699e5a6735aae6bc296f48b1463.tar.gz
DEBUG:python:Writing Docker layers files to /tmp/.singularity-layers.c1DXDRgy
INFO:python:Writing file /tmp/.singularity-layers.c1DXDRgy with mode w.
INFO:python:Writing file /tmp/.singularity-layers.c1DXDRgy with mode a.
INFO:python:*** FINISHING DOCKER IMPORT PYTHON PORTION ****

Exploding layer: sha256:ad1a736aa04951e504861c84b577c102f3950fcd329bbda7347bb2d88d2300d7.tar.gz
Exploding layer: sha256:45a2e645736c4c66ef34acce2407ded21f7a9b231199d3b92d6c9776df264729.tar.gz
Exploding layer: sha256:cebfcb455b0f307f2cc86b3c54677c4a78279699e5a6735aae6bc296f48b1463.tar.gz
VERBOSE [U=0,P=21484]      message_init()                            Set messagelevel to: 5
VERBOSE [U=0,P=21484]      singularity_config_parse()                Initialize configuration file: /usr/local/etc/singularity/singularity.conf
DEBUG   [U=0,P=21484]      singularity_config_parse()                Starting parse of configuration file /usr/local/etc/singularity/singularity.conf
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key enable overlay = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key container dir = '/var/lib/singularity/mnt'
VERBOSE [U=0,P=21484]      singularity_config_parse()                Got config key singularity user = 'singularity'
DEBUG   [U=0,P=21484]      singularity_config_parse()                Finished parsing configuration file '/usr/local/etc/singularity/singularity.conf'
VERBOSE [U=0,P=21484]      singularity_suid_init()                   Running SUID program workflow
VERBOSE [U=0,P=21484]      singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=0,P=21484]      singularity_suid_init()                   Checking configuration file is properly owned by root
VERBOSE [U=0,P=21484]      singularity_suid_init()                   Checking if singularity.conf allows us to run as suid
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow setuid, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value allow setuid='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow setuid, yes) = 1
VERBOSE [U=0,P=21484]      singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'WRITABLE' = '1'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(WRITABLE, 1) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(libexecdir, /usr/local/libexec) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'COMMAND' = 'shell'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(COMMAND, shell) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'ROOTFS' = '/tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(ROOTFS, /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'VERSION' = '2.2.99'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(version, 2.2.99) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(localstatedir, /usr/local/var) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'CONTENTS' = '/tmp/.singularity-layers.c1DXDRgy'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(CONTENTS, /tmp/.singularity-layers.c1DXDRgy) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /usr/local/etc) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'CONTAINER' = 'docker://blacklabelops/centos'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(CONTAINER, docker://blacklabelops/centos) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'IMAGE' = '/tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(IMAGE, /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos) = 0
VERBOSE [U=0,P=21484]      singularity_registry_set()                Adding value to registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.cZQkGQt7'
DEBUG   [U=0,P=21484]      singularity_registry_set()                Returning singularity_registry_set(SESSIONDIR, /tmp/.singularity-runtime.cZQkGQt7) = 0
DEBUG   [U=0,P=21484]      singularity_priv_init()                   Initializing user info
DEBUG   [U=0,P=21484]      singularity_priv_init()                   Set the calling user's username to: root
DEBUG   [U=0,P=21484]      singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=0,P=21484]      singularity_priv_init()                   Obtaining home directory
VERBOSE [U=0,P=21484]      singularity_priv_init()                   Set home (via getpwuid()) to: /root
VERBOSE [U=0,P=21484]      singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
VERBOSE [U=0,P=21484]      singularity_priv_userns()                 Not virtualizing USER namespace: running as root
DEBUG   [U=0,P=21484]      singularity_priv_userns()                 Returning singularity_priv_init(void)
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_registry_get()                Retriving value from registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.cZQkGQt7'
DEBUG   [U=0,P=21484]      singularity_sessiondir()                  Got SINGULARITY_SESSIONDIR: (null)
VERBOSE [U=0,P=21484]      singularity_sessiondir()                  Creating session directory: /tmp/.singularity-runtime.cZQkGQt7
DEBUG   [U=0,P=21484]      singularity_sessiondir()                  Opening sessiondir file descriptor
DEBUG   [U=0,P=21484]      singularity_sessiondir()                  Setting shared flock() on session directory
DEBUG   [U=0,P=21484]      singularity_sessiondir()                  Spawning a session directory cleanup process
VERBOSE [U=0,P=31979]      message_init()                            Set messagelevel to: 5
DEBUG   [U=0,P=31979]      envar_path()                              Checking environment variable is valid path: 'SINGULARITY_CLEANDIR'
VERBOSE [U=0,P=31979]      envar()                                   Checking input from environment: 'SINGULARITY_CLEANDIR'
DEBUG   [U=0,P=31979]      envar()                                   Checking environment variable is defined: SINGULARITY_CLEANDIR
DEBUG   [U=0,P=31979]      envar()                                   Checking environment variable length (<= 4096): SINGULARITY_CLEANDIR
DEBUG   [U=0,P=31979]      envar()                                   Checking environment variable has allowed characters: SINGULARITY_CLEANDIR
VERBOSE [U=0,P=31979]      envar()                                   Obtained input from environment 'SINGULARITY_CLEANDIR' = '/tmp/.singularity-runtime.cZQkGQt7'
DEBUG   [U=0,P=31979]      main()                                    Starting cleanup process
DEBUG   [U=0,P=31979]      main()                                    Opening cleandir file descriptor
DEBUG   [U=0,P=31979]      main()                                    Opening cleandir_lock file descriptor
VERBOSE [U=0,P=31979]      main()                                    Daemonizing cleandir cleanup process
DEBUG   [U=0,P=21484]      singularity_registry_get()                Retriving value from registry: 'IMAGE' = '/tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos'
DEBUG   [U=0,P=21484]      singularity_registry_get()                Retriving value from registry: 'WRITABLE' = '1'
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       No configuration entry found for 'limit container owners'; returning default value 'NULL'
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       No configuration entry found for 'limit container paths'; returning default value 'NULL'
DEBUG   [U=0,P=21484]      singularity_image_open()                  Opening file descriptor to image: /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos
DEBUG   [U=0,P=21484]      singularity_registry_get()                Retriving value from registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.cZQkGQt7'
DEBUG   [U=0,P=21484]      singularity_runtime_ns()                  Calling: _singularity_runtime_ns_pid()
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow pid ns, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value allow pid ns='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow pid ns, yes) = 1
VERBOSE [U=0,P=21484]      singularity_runtime_ns_pid()              Not virtualizing PID namespace on user request
DEBUG   [U=0,P=21484]      singularity_runtime_ns()                  Calling: _singularity_runtime_ns_ipc()
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow ipc ns, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       No configuration entry found for 'allow ipc ns'; returning default value 'yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow ipc ns, yes) = 1
VERBOSE [U=0,P=21484]      singularity_runtime_ns_ipc()              Not virtualizing IPC namespace on user request
DEBUG   [U=0,P=21484]      singularity_runtime_ns()                  Calling: _singularity_runtime_ns_mnt()
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount slave, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount slave='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount slave, yes) = 1
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_runtime_ns_mnt()              Virtualizing FS namespace
DEBUG   [U=0,P=21484]      singularity_runtime_ns_mnt()              Virtualizing mount namespace
DEBUG   [U=0,P=21484]      singularity_runtime_ns_mnt()              Making mounts slave
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value max loop devices='256'
DEBUG   [U=0,P=21484]      singularity_image_bind()                  Entered singularity_image_bind()
DEBUG   [U=0,P=21484]      singularity_image_bind()                  Converting max_loop_devs_string to int: '256'
DEBUG   [U=0,P=21484]      singularity_image_bind()                  Converted max_loop_devs_string to int: '256' -> 256
DEBUG   [U=0,P=21484]      singularity_image_bind()                  Checking if this image has been properly opened
DEBUG   [U=0,P=21484]      singularity_image_bind()                  Checking if image is valid file
VERBOSE [U=0,P=21484]      singularity_image_bind()                  Skipping image bind, container is not a file
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value container dir='/var/lib/singularity/mnt'
DEBUG   [U=0,P=21484]      singularity_runtime_rootfs()              Setting container directory to: /var/lib/singularity/mnt/source
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_runtime_rootfs()              Creating top level source mount directory to: /var/lib/singularity/mnt/source
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_image_mount()                 Checking what kind of image we are mounting
DEBUG   [U=0,P=21484]      singularity_image_mount_squashfs_check()  Image does not appear to be of type '.sqsh': /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos
DEBUG   [U=0,P=21484]      singularity_image_mount_dir_check()       Checking if image is a directory
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_image_mount_dir_mount()       Mounting container directory /tmp/.singularity-runtime.cZQkGQt7/container/blacklabelops/centos->/var/lib/singularity/mnt/source
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      action_ready()                            Checking if container is valid at: /var/lib/singularity/mnt/source
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value container dir='/var/lib/singularity/mnt'
DEBUG   [U=0,P=21484]      singularity_runtime_overlayfs()           Checking if overlayfs should be used
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(enable overlay, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value enable overlay='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(enable overlay, yes) = 1
DEBUG   [U=0,P=21484]      singularity_registry_get()                Retriving value from registry: 'WRITABLE' = '1'
VERBOSE [U=0,P=21484]      singularity_runtime_overlayfs()           Not enabling overlayFS, image mounted writablable
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_runtime_overlayfs()           Creating mount_final directory: /var/lib/singularity/mnt/final
VERBOSE [U=0,P=21484]      singularity_runtime_overlayfs()           Binding the ROOTFS_SOURCE to OVERLAY_FINAL (/var/lib/singularity/mnt/source->/var/lib/singularity/mnt/final)
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_overlayfs()           Updating the containerdir to: /var/lib/singularity/mnt/final
VERBOSE [U=0,P=21484]      singularity_runtime_mounts()              Running all mount components
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount hostfs, no)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount hostfs='no'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount hostfs, no) = 0
DEBUG   [U=0,P=21484]      singularity_runtime_mount_hostfs()        Not mounting host file systems per configuration
DEBUG   [U=0,P=21484]      singularity_runtime_mount_binds()         Checking configuration file for 'bind path'
VERBOSE [U=0,P=21484]      singularity_runtime_mount_binds()         Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=0,P=21484]      singularity_runtime_mount_binds()         Checking if bind point is already mounted: /etc/hosts
DEBUG   [U=0,P=21484]      check_mounted()                           Opening /proc/mounts
DEBUG   [U=0,P=21484]      check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_binds()         Binding '/etc/hosts' to '/var/lib/singularity/mnt/final//etc/hosts'
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount proc'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount proc, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount proc='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount proc, yes) = 1
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_kernelfs()      Mounting /proc
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount sys'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount sys, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount sys='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount sys, yes) = 1
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_kernelfs()      Mounting /sys
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
DEBUG   [U=0,P=21484]      singularity_runtime_mount_dev()           Checking configuration file for 'mount dev'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount dev, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount dev, yes) = 1
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_dev()           Bind mounting /dev
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount tmp, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount tmp='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount tmp, yes) = 1
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_tmp()           Mounting directory: /tmp
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_tmp()           Mounting directory: /var/tmp
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount home, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value mount home='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount home, yes) = 1
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Checking that home directry is configured: /root
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Identifying the base directory of homedir: /root
DEBUG   [U=0,P=21484]      basedir()                                 Obtaining basedir for: /root
DEBUG   [U=0,P=21484]      basedir()                                 Iterating basedir: /root
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Checking if home directory is already mounted: /root
DEBUG   [U=0,P=21484]      check_mounted()                           Opening /proc/mounts
DEBUG   [U=0,P=21484]      check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Creating directory to stage tmpdir home: /tmp/.singularity-runtime.cZQkGQt7/root
DEBUG   [U=0,P=21484]      s_mkpath()                                Creating directory: /tmp/.singularity-runtime.cZQkGQt7/root
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Configuring the source of the home directory
VERBOSE [U=0,P=21484]      singularity_runtime_mount_home()          Setting home directory source from singularity_priv_homedir()
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Set home directory source to: /root
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Checking to make sure that the home directory exists: /root
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Checking ownership of physical home directory: /root
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_home()          Mounting home directory source to stage: /root->/tmp/.singularity-runtime.cZQkGQt7/root
DEBUG   [U=0,P=21484]      singularity_runtime_mount_home()          Remounting home directory with necessary options: /root
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_mount_home()          Mounting home directory base into container: /tmp/.singularity-runtime.cZQkGQt7/root->/var/lib/singularity/mnt/final/root
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_runtime_mount_userbinds()     Checking for environment variable 'SINGULARITY_BINDPATH'
DEBUG   [U=0,P=21484]      singularity_runtime_mount_userbinds()     No user bind mounts specified.
DEBUG   [U=0,P=21484]      singularity_runtime_mount_scratch()       Getting SINGULARITY_SCRATCHDIR from environment
DEBUG   [U=0,P=21484]      singularity_runtime_mount_scratch()       Not mounting scratch directory: Not requested
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Checking to see if we should mount current working directory
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Getting current working directory
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Checking if current directory exists in container
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Container does not have the directory: /home/vanessa
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Checking for contain option
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Checking if CWD is already mounted: /home/vanessa
DEBUG   [U=0,P=21484]      check_mounted()                           Opening /proc/mounts
DEBUG   [U=0,P=21484]      check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Checking if cwd is in an operating system directory
DEBUG   [U=0,P=21484]      singularity_runtime_mount_cwd()           Checking if overlay is enabled
VERBOSE [U=0,P=21484]      singularity_runtime_mount_cwd()           Not mounting current directory: overlay is not enabled and directory does not exist in container: /var/lib/singularity/mnt/final/home/vanessa
VERBOSE [U=0,P=21484]      singularity_runtime_files()               Running file components
DEBUG   [U=0,P=21484]      singularity_runtime_files_passwd()        Called singularity_file_passwd_create()
VERBOSE [U=0,P=21484]      singularity_runtime_files_passwd()        Not updating passwd file, running as root!
DEBUG   [U=0,P=21484]      singularity_runtime_files_group()         Called singularity_file_group_create()
VERBOSE [U=0,P=21484]      singularity_runtime_files_group()         Not updating group file, running as root!
DEBUG   [U=0,P=21484]      singularity_runtime_files_resolvconf()    Checking configuration option
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config resolv_conf, yes)
DEBUG   [U=0,P=21484]      singularity_config_get_value_impl()       Returning configuration value config resolv_conf='yes'
DEBUG   [U=0,P=21484]      singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config resolv_conf, yes) = 1
DEBUG   [U=0,P=21484]      container_file_bind()                     Called file_bind(/etc/resolv.conf, /etc/resolv.conf()
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      container_file_bind()                     Binding file '/etc/resolv.conf' to '/var/lib/singularity/mnt/final/etc/resolv.conf'
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_enter()               Containing all rootfs components
DEBUG   [U=0,P=21484]      singularity_priv_escalate()               Running as root, not changing privileges
VERBOSE [U=0,P=21484]      singularity_runtime_enter_chroot()        Entering container file system root: /var/lib/singularity/mnt/final
DEBUG   [U=0,P=21484]      singularity_priv_drop()                   Running as root, not changing privileges
DEBUG   [U=0,P=21484]      singularity_runtime_enter_chroot()        Changing dir to '/' within the new root
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Counting environment vars
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Transposing SINGULARITYENV variables ('32' total)
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Cleaning SINGULARITY_* envars
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_WRITABLE
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_libexecdir
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_COMMAND
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_MESSAGELEVEL
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_ROOTFS
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_version
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_localstatedir
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTENTS
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_sysconfdir
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTAINER
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_IMAGE
DEBUG   [U=0,P=21484]      singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_SESSIONDIR
DEBUG   [U=0,P=21484]      singularity_priv_drop_perm()              Called singularity_priv_drop_perm(void)
VERBOSE [U=0,P=21484]      singularity_priv_drop_perm()              Calling user is root, no privileges to drop
VERBOSE [U=0,P=21484]      main()                                    Current directory is not available within container, landing in home
DEBUG   [U=0,P=21484]      singularity_registry_get()                Retriving value from registry: 'COMMAND' = 'shell'
INFO    [U=0,P=21484]      action_shell()                            Singularity: Invoking an interactive shell within container...

DEBUG   [U=0,P=21484]      action_shell()                            Exec'ing /.singularity.d/actions/shell
Singularity centos:/root>
abdulrahmanazab commented 7 years ago

@vsoch @gmkurtzer a case to reproduce: sudo singularity --debug shell -w docker://poldracklab/mriqc:latest Those files should be executable, but they are not: -rw-rw-r-- 1 root root 22 Mar 22 04:06 /usr/bin/run_dfcheck -rw-rw-r-- 1 root root 70 Mar 22 04:06 /usr/bin/run_mriqc -rw-rw-r-- 1 root root 26 Mar 22 04:06 /usr/bin/run_mriqc_plot -rw-rw-r-- 1 root root 86 Mar 22 04:06 /usr/bin/run_tests

vsoch commented 7 years ago

Are you using latest development? I can't reproduce:

root@vanessa-ThinkPad-T450s:/root# ls -l /usr/bin/run*
-rwxr-xr-x 1 root root 17975 Dec 28  2014 /usr/bin/run-mailcap
-rwxrwxr-x 1 root root    22 Mar 22 04:06 /usr/bin/run_dfcheck
-rwxrwxr-x 1 root root    70 Mar 22 04:06 /usr/bin/run_mriqc
-rwxrwxr-x 1 root root    26 Mar 22 04:06 /usr/bin/run_mriqc_plot
-rwxrwxr-x 1 root root    86 Mar 22 04:06 /usr/bin/run_tests
-rwxr-xr-x 1 root root 31336 Mar 14  2015 /usr/bin/runcon

That is one FAT image!! He needs a Docker diet, pronto!!

abdulrahmanazab commented 7 years ago

@vsoch Yeah, I was trying 2.2.1. Now with the dev (2.2.99) I get: ERROR:python:Cannot write to /root/.singularity/docker/sha256:cdb4d5376bdead059bc547128992c4a4e2f601a606acbe1447fb00ebeb725fe4.tar.gz.tmp._wVM2l, exiting ERROR:python:Error downloading https://index.docker.io/v2/poldracklab/mriqc/blobs/sha256:cdb4d5376bdead059bc547128992c4a4e2f601a606acbe1447fb00ebeb725fe4. Do you have permission to write to /root/.singularity/docker?

when I run: sudo singularity --debug import poldracklab-mriqc.img docker://poldracklab/mriqc:lates

This is probably because I have a very little space on /root but why in the world does it try to pull the layers inside /root/.singularity/docker when I already specify an image to write in?

abdulrahmanazab commented 7 years ago

@vsoch It works now after I tried it somewhere else with a bigger /root. But the use of /root/.singularity/docker to store layers even when you specify a target image is not nice.

Anyways, would this be fixable without sudo and the --writable option?

vsoch commented 7 years ago

Yes I got the same error when my hard drive was running out of space (because this image is so fat!) Is there a reason you can't use create then import instead of shell? We will eventually be combining these two functionalities into a "pull" command, but for now it's like:

vanessa@vanessa-ThinkPad-T450s:~/Desktop$ singularity create -s 6000 cafe.img
Initializing Singularity image subsystem
Opening image file: cafe.img
Creating 6000MiB image
Binding image to loop
Creating file system within image
Image is done: cafe.img
vanessa@vanessa-ThinkPad-T450s:~/Desktop$ singularity import cafe.img docker://poldracklab/mriqc:latest
Cache folder set to /home/vanessa/.singularity/docker
Importing: base Singularity environment
Importing: /home/vanessa/.singularity/docker/sha256:357ea8c3d80bc25792e010facfc98aee5972ebc47e290eb0d5aea3671a901cab.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:fb95ab5480340193a679f58c726e208ce0a14a881cc48f3ec6b68490ca768abe.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:dfc27603e46bd3d4ee9b3e08f58b6a6b577c6cb5a6cb31ddc38937117f8d847f.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:f7e997984e865e5dc9a44099240240b68e5315cfe2fa33fe0536c4a69bb384a9.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:d16185ca875cfc73da65eb0621e3cb38544a728079f7b6aa3f9c0cc2eaf05ef2.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:ebf965b3a12075202089704d5dbdcbad376142df41a3c9e0e52a2ce3c205cb93.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:1b1839d10fc8ed13ebedae248e8dfec1ff9ab0dabe0ef66407daff379781d9bb.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:cdb4d5376bdead059bc547128992c4a4e2f601a606acbe1447fb00ebeb725fe4.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:397afeac127195ad688c8217771a958b5314d37b9950c23e5bf3ec11fd617383.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:21ffec20c4fbca9ec0da6443ecb6ee19532cf6bad67e718ba731ac907f994d6b.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:349f16732898335edd78342081fdc86ffbe53399383594151478fde06a51772d.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:02975f5cac2a64faa614c07688abb109ed205f992f7b4524ca564f8a54e3352d.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:62fb7fa8a829f91dfba3457433485a3e93f574e1702d9be2089fc11a3845ca6d.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:1d25a9af101b25f769c0c613b0806404d96e3fc7787cb9caf1e89053695fa922.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:01b45f97a2d05cade617aa710fcbf373c0758964432d99bec5e30d077936c0bb.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:c251c4d6043b7ddea5ffacc1329fd2ea72988ae17ecbb387f9344d0ee750cc68.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:8a822fbd478b67df01840494313cb0eaf7a0e788a46b91f9e7e8d31e99636f06.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:e40ec34eb96f67941b012f19f8a6cd9e788e099e16db61ebf650fc59b7668373.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:42f53768bf1b0d14d6e0d53b24e5f5d51c618816ec5e279c4e7090186717193c.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Importing: /home/vanessa/.singularity/docker/sha256:e8474564357be0bc18d544bb729a337c8313a29308901349ead0e8ae98466f82.tar.gz
vanessa@vanessa-ThinkPad-T450s:~/Desktop$ singularity shell cafe.img 
Singularity: Invoking an interactive shell within container...

$ ls -l /usr/bin/run*
-rwxr-xr-x 1 root root 17975 Dec 28  2014 /usr/bin/run-mailcap
-rwxrwxr-x 1 root root    22 Mar 22 04:06 /usr/bin/run_dfcheck
-rwxrwxr-x 1 root root    70 Mar 22 04:06 /usr/bin/run_mriqc
-rwxrwxr-x 1 root root    26 Mar 22 04:06 /usr/bin/run_mriqc_plot
-rwxrwxr-x 1 root root    86 Mar 22 04:06 /usr/bin/run_tests
-rwxr-xr-x 1 root root 31336 Mar 14  2015 /usr/bin/runcon

No sudo required for the above :)

abdulrahmanazab commented 7 years ago

@vsoch Yeah, it works! So singularity import doesn't need sudo? cool!

But why singularity bootstrap still need it then?

And BTW! I saw this in the output of the import: Cache folder set to <my-home>/.singularity/docker

Which means that even when you import to an image that you created, the layers are still cached to the home directory. Is this necessary? If yes, can I change the cache folder path (in case I don't have space in $HOME)?

vsoch commented 7 years ago

haha, well what looks sort of similar / the same has different underlying things going on. Anything that doesn't require sudo is just obtaining layers and dumping them into the image, with their permissions as is. If you look in the bootstraps scripts folder, there is often a lot of functionality that goes beyond just dumping layers into an image, like stuff that happens on the host in setup, or things that would require sudo in any section, and I think the idea of bootstrap is to fit that use case. But I think you are right that we could have a subset / kind of bootstrap that doesn't require the extra bits (probably one using Docker and just defining environment vars and not much else), and basically fulfills the same kind of things as the current import (without sudo).

I think our first idea is to wrap the above two commands into one "pull" which will create an image and dump layers into it, in one go. We will definitely think more about the "non sudo" bootstrap version, and how to best integrate it though.

abdulrahmanazab commented 7 years ago

@vsoch Thanks :) Another thing to investigate. I noticed that $PATH is not the same. See:

[azab@invivo singularity-img]$ singularity shell poldracklab-mriqc4.img
Singularity: Invoking an interactive shell within container...

bash: /software/etc/profile/bashrc: No such file or directory
azab@invivo:~/singularity-img$ echo $PATH
/opt/c3d:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/ants:/usr/lib/fsl/5.0:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Now when I run docker:

[azab@invivo singularity-img]$ docker run -u 238869:173131 --rm -it --entrypoint=/bin/bash poldracklab/mriqc
I have no name!@ac75c3605b48:/scratch$ echo $PATH
/usr/local/miniconda/bin:/opt/c3d:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/ants:/usr/lib/fsl/5.0:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

/usr/local/miniconda/bin is missing with the singularity container

vsoch commented 7 years ago

hey @abdulrahmanazab apologies for delay - you found a bug! I just fixed it - https://github.com/singularityware/singularity/pull/543 the issue was using the last in the list of history for the environment (or other config) and we needed to use the first. So the path that you see was the first (original) that didn't have the miniconda export. With this fix, it should be equivalent. Good catch! :)

abdulrahmanazab commented 7 years ago

@vsoch Great! thanks :) I see that you are using the mriqc image for tests ;) When would it be merged?

vsoch commented 7 years ago

When I have time to fly from east to west coast, get home, write the node code, do a PR, have @gmkurtzer review, and then merge :) So aim for tomorrow to be safe!

vsoch commented 7 years ago

This should be good !