apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

ERROR: Could not remove run directory (...) Directory not empty #662

Closed chrisgorgo closed 6 years ago

chrisgorgo commented 7 years ago

I'm getting the following error at the end of execution:

[root@efa4dfbd5360 /]# singularity run docker://poldracklab/mriqc:0.9.3
Cache folder set to /root/.singularity/docker
Extracting /root/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Downloading layer sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83
Extracting /root/.singularity/docker/sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83.tar.gz
Downloading layer sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439
Extracting /root/.singularity/docker/sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439.tar.gz
Downloading layer sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f
Extracting /root/.singularity/docker/sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f.tar.gz
Downloading layer sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac
Extracting /root/.singularity/docker/sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac.tar.gz
Downloading layer sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0
Extracting /root/.singularity/docker/sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0.tar.gz
Downloading layer sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5
Extracting /root/.singularity/docker/sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5.tar.gz
Downloading layer sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d
Extracting /root/.singularity/docker/sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d.tar.gz
Downloading layer sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810
Extracting /root/.singularity/docker/sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810.tar.gz
Downloading layer sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274
Extracting /root/.singularity/docker/sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274.tar.gz
Downloading layer sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb
Extracting /root/.singularity/docker/sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb.tar.gz
Downloading layer sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39
Extracting /root/.singularity/docker/sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39.tar.gz
Downloading layer sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa
Extracting /root/.singularity/docker/sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa.tar.gz
Downloading layer sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351
Extracting /root/.singularity/docker/sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351.tar.gz
Downloading layer sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163
Extracting /root/.singularity/docker/sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163.tar.gz
Downloading layer sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef
Extracting /root/.singularity/docker/sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef.tar.gz
Downloading layer sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d
Extracting /root/.singularity/docker/sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d.tar.gz
Downloading layer sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3
Extracting /root/.singularity/docker/sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3.tar.gz
Downloading layer sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8
Extracting /root/.singularity/docker/sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8.tar.gz
Downloading layer sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6
Extracting /root/.singularity/docker/sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6.tar.gz
Downloading layer sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608
Extracting /root/.singularity/docker/sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608.tar.gz
Downloading layer sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2
Extracting /root/.singularity/docker/sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2.tar.gz
Downloading layer sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714
Extracting /root/.singularity/docker/sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714.tar.gz
Downloading layer sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787
Extracting /root/.singularity/docker/sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787.tar.gz
Adding Docker ENTRYPOINT as Singularity runscript...
/usr/local/miniconda/bin/mriqc
usage: mriqc [-h] [-v]
             [--participant_label [PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]]
             [-m [{T1w,bold,T2w} [{T1w,bold,T2w} ...]]] [-s SESSION_ID]
             [-r RUN_ID] [--nthreads NTHREADS] [--n_procs N_PROCS]
             [--mem_gb MEM_GB] [--write-graph] [--dry-run]
             [--use-plugin USE_PLUGIN] [--ica] [--testing] [--profile]
             [--hmc-afni] [--hmc-fsl] [-f] [--fft-spikes-detector]
             [-w WORK_DIR] [--report-dir REPORT_DIR] [--verbose-reports]
             [--ants-nthreads ANTS_NTHREADS] [--ants-settings ANTS_SETTINGS]
             [--deoblique] [--despike] [--start-idx START_IDX]
             [--stop-idx STOP_IDX] [--correct-slice-timing]
             bids_dir output_dir {participant,group} [{participant,group} ...]
mriqc: error: the following arguments are required: bids_dir, output_dir, analysis_level
Singularity: sexec (U=0,P=61)> Could not remove run directory /tmp/singularity-container_dir.KJo66K53/poldracklab/mriqc:0.9.3: Directory not empty

ERROR  : Could not remove run directory /tmp/singularity-container_dir.KJo66K53/poldracklab/mriqc:0.9.3: Directory not empty
vsoch commented 7 years ago

Are you using development branch?

chrisgorgo commented 7 years ago

I was not (sorry for that). I've tried again with 945c6ee343a1e6101e22396a90dfdb5944f442b6 (most current on development branch) and I get the same error:

[root@e4fb0252c326 /]# singularity run docker://bids/mriqc:0.9.3
Docker image path: index.docker.io/bids/mriqc:0.9.3
Cache folder set to /root/.singularity/docker
[25/25] |===================================| 100.0%
Creating container runtime...
Singularity: action-suid (U=0,P=27)> USER=root, IMAGE='mriqc:0.9.3', COMMAND='run'

Fontconfig warning: ignoring C.UTF-8: not a valid language tag
usage: mriqc [-h] [-v]
             [--participant_label [PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]]
             [-m [{T1w,bold,T2w} [{T1w,bold,T2w} ...]]] [-s SESSION_ID]
             [-r RUN_ID] [--nthreads NTHREADS] [--n_procs N_PROCS]
             [--mem_gb MEM_GB] [--write-graph] [--dry-run]
             [--use-plugin USE_PLUGIN] [--ica] [--testing] [--profile]
             [--hmc-afni] [--hmc-fsl] [-f] [--fft-spikes-detector]
             [-w WORK_DIR] [--report-dir REPORT_DIR] [--verbose-reports]
             [--ants-nthreads ANTS_NTHREADS] [--ants-settings ANTS_SETTINGS]
             [--deoblique] [--despike] [--start-idx START_IDX]
             [--stop-idx STOP_IDX] [--correct-slice-timing]
             bids_dir output_dir {participant,group} [{participant,group} ...]
mriqc: error: the following arguments are required: bids_dir, output_dir, analysis_level
[root@e4fb0252c326 /]# Singularity: singularity: cleanupd (U=0,P=100511)> Failed removing file: /tmp/.singularity-runtime.e9QOcU07/container/bids/mriqc:0.9.3/libwebp-0.5.2-linux-x86-64/include/webp

Singularity: singularity: cleanupd (U=0,P=100511)> Could not remove directory /tmp/.singularity-runtime.e9QOcU07: Directory not empty

Singularity: singularity: cleanupd (U=0,P=100511)> Retval = 255
vsoch commented 7 years ago

hey @chrisfilo ! No worries about the version. I think in this case, running as root would mean that home is in /root, and then the container's home is mounted as such, but that is also where the application lives... and there must be some permission error cleaning up the temporary directory. I just tried this as a (non root) user and it seems to work ok:

vanessa@vanessa-ThinkPad-T450s:~$ singularity run docker://bids/mriqc:0.9.3
Docker image path: index.docker.io/bids/mriqc:0.9.3
Cache folder set to /home/vanessa/.singularity/docker
[25/25] |                                   | 100.0% 
Importing: base Singularity environment
Exploding layer: sha256:243eedf190b75e91c01f330ff21c77f4c6dc385d7756dda9e7f39741ad4f4071.tar.gz
Exploding layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Exploding layer: sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83.tar.gz
Exploding layer: sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439.tar.gz
Exploding layer: sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f.tar.gz
Exploding layer: sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac.tar.gz
Exploding layer: sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0.tar.gz
Exploding layer: sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5.tar.gz
Exploding layer: sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d.tar.gz
Exploding layer: sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810.tar.gz
Exploding layer: sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274.tar.gz
Exploding layer: sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb.tar.gz
Exploding layer: sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39.tar.gz
Exploding layer: sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa.tar.gz
Exploding layer: sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351.tar.gz
Exploding layer: sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163.tar.gz
Exploding layer: sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef.tar.gz
Exploding layer: sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d.tar.gz
Exploding layer: sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3.tar.gz
Exploding layer: sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8.tar.gz
Exploding layer: sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6.tar.gz
Exploding layer: sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608.tar.gz
Exploding layer: sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2.tar.gz
Exploding layer: sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714.tar.gz
Exploding layer: sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787.tar.gz
Exploding layer: sha256:2bf1a0cf6f2a1259591399f9afe1956e03128633732c390aa0889af805e1b9a4.tar.gz
/usr/local/miniconda/lib/python3.6/site-packages/sklearn/externals/joblib/_multiprocessing_helpers.py:28: UserWarning: [Errno 38] Function not implemented.  joblib will operate in serial mode
  warnings.warn('%s.  joblib will operate in serial mode' % (e,))
Fontconfig warning: ignoring C.UTF-8: not a valid language tag
usage: mriqc [-h] [-v]
             [--participant_label [PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]]
             [-m [{T1w,bold,T2w} [{T1w,bold,T2w} ...]]] [-s SESSION_ID]
             [-r RUN_ID] [--nthreads NTHREADS] [--n_procs N_PROCS]
             [--mem_gb MEM_GB] [--write-graph] [--dry-run]
             [--use-plugin USE_PLUGIN] [--ica] [--testing] [--profile]
             [--hmc-afni] [--hmc-fsl] [-f] [--fft-spikes-detector]
             [-w WORK_DIR] [--report-dir REPORT_DIR] [--verbose-reports]
             [--ants-nthreads ANTS_NTHREADS] [--ants-settings ANTS_SETTINGS]
             [--deoblique] [--despike] [--start-idx START_IDX]
             [--stop-idx STOP_IDX] [--correct-slice-timing]
             bids_dir output_dir {participant,group} [{participant,group} ...]
mriqc: error: the following arguments are required: bids_dir, output_dir, analysis_level

some things to try:

Anyway - whatever you try - please post the commands and debug output so I can help to resolve. Ty!

chrisgorgo commented 7 years ago

I've tried with non-root user. Same result:

[newuser@febd2ce0fb13 ~]$ singularity exec docker://bids/mriqc:0.9.3 env
Docker image path: index.docker.io/bids/mriqc:0.9.3
Cache folder set to /home/newuser/.singularity/docker
[25/25] |===================================| 100.0%
Creating container runtime...
Singularity: action-suid (U=500,P=31)> USER=newuser, IMAGE='mriqc:0.9.3', COMMAND='exec'

LESSOPEN=||/usr/bin/lesspipe.sh %s
POSSUMDIR=/usr/share/fsl/5.0
AFNI_TTATLAS_DATASET=/usr/share/afni/atlases
AFNI_PLUGINPATH=/usr/lib/afni/plugins
USER=newuser
HOSTNAME=febd2ce0fb13
SHLVL=1
LD_LIBRARY_PATH=/usr/lib/fsl/5.0::/.singularity.d/libs
HOME=/home/newuser
PS1=Singularity>
G_BROKEN_FILENAMES=1
FSLDIR=/usr/share/fsl/5.0
AFNI_IMSAVE_WARNINGS=NO
ANTSPATH=/opt/ants
FSLMULTIFILEQUIT=TRUE
FSLTCLSH=/usr/bin/tclsh
TERM=xterm
PATH=/usr/local/miniconda/bin:/opt/ants:/usr/lib/fsl/5.0:/usr/lib/afni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
FSLOUTPUTTYPE=NIFTI_GZ
LANG=C.UTF-8
AFNI_MODELPATH=/usr/lib/afni/models
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
FSLWISH=/usr/bin/wish
CRN_SHARED_DATA=/niworkflows_data
SINGULARITY_CONTAINER=mriqc:0.9.3
PWD=/home/newuser
LC_ALL=C.UTF-8
HISTFILE=
SINGULARITY_NAME=mriqc:0.9.3
[newuser@febd2ce0fb13 ~]$ Singularity: singularity: cleanupd (U=500,P=100718)> Failed removing file: /tmp/.singularity-runtime.2CKUBwBh/container/bids/mriqc:0.9.3/libwebp-0.5.2-linux-x86-64/include/webp

Singularity: singularity: cleanupd (U=500,P=100718)> Could not remove directory /tmp/.singularity-runtime.2CKUBwBh: Directory not empty

Singularity: singularity: cleanupd (U=500,P=100718)> Retval = 255

Here's the content of the relevant tmp folder:

[newuser@febd2ce0fb13 ~]$ ls -al /tmp/.singularity-runtime.2CKUBwBh/container/bids/mriqc\:0.9.3/
total 2236
drwx------ 23 newuser newuser    4096 May  9 19:52 .
drwx------  3 newuser newuser    4096 May  9 19:48 ..
drwx------  2 newuser newuser    4096 Dec 13 20:24 bin
drwx------  2 newuser newuser    4096 Apr 12  2016 boot
drwx------  2 newuser newuser    4096 Dec 13 20:23 dev
lrwxrwxrwx  1 newuser newuser      36 May  9 19:49 environment -> .singularity.d/env/90-environment.sh
drwx------ 60 newuser newuser    4096 Dec 13 20:24 etc
lrwxrwxrwx  1 newuser newuser      27 May  9 19:49 .exec -> .singularity.d/actions/exec
drwx------  2 newuser newuser    4096 Apr 12  2016 home
drwx------  8 newuser newuser    4096 Sep 13  2015 lib
drwx------  2 newuser newuser    4096 Dec 13 20:24 lib64
drwx------  6 newuser newuser    4096 May  9 19:51 libwebp-0.5.2-linux-x86-64
-rw-r--r--  1 newuser newuser 2185345 Apr 23 18:15 libwebp-0.5.2-linux-x86-64.tar.gz
drwx------  2 newuser newuser    4096 Dec 13 20:23 media
drwx------  2 newuser newuser    4096 Dec 13 20:23 mnt
drwx------  3 newuser newuser    4096 Apr 23 18:21 niworkflows_data
drwx------  3 newuser newuser    4096 Dec 13 20:23 opt
drwx------  2 newuser newuser    4096 Apr 12  2016 proc
-rw-rw-r--  1 newuser newuser     213 Apr 23 18:07 requirements.txt
drwx------  8 newuser newuser    4096 Dec 13 20:24 root
drwx------  5 newuser newuser    4096 Dec 13 20:24 run
lrwxrwxrwx  1 newuser newuser      26 May  9 19:49 .run -> .singularity.d/actions/run
drwx------  2 newuser newuser    4096 Dec 13 20:24 sbin
lrwxrwxrwx  1 newuser newuser      28 May  9 19:49 .shell -> .singularity.d/actions/shell
lrwxrwxrwx  1 newuser newuser      24 May  9 19:49 singularity -> .singularity.d/runscript
drwx------  5 newuser newuser    4096 May  9 19:52 .singularity.d
drwx------  3 newuser newuser    4096 Feb  5  2016 sys
drwx------  3 newuser newuser    4096 Dec 13 20:24 tmp
drwx------ 11 newuser newuser    4096 Dec 13 20:23 usr
drwx------ 11 newuser newuser    4096 Dec 13 20:24 var
-rw-rw-r--  1 newuser newuser       6 May  9 03:37 version

I don't get this error when running with --debug, but I don't think singularity is attempting to delete the folder:

[newuser@0e726194b97a ~]$ singularity --debug exec docker://bids/mriqc:0.9.3 env
Enabling debugging
Ending argument loop
Singularity version: 2.2.99-HEAD.g945c6ee
Exec'ing: /usr/local/libexec/singularity/cli/exec.exec
Evaluating args: 'docker://bids/mriqc:0.9.3 env'
VERBOSE2 SINGULARITY_COMMAND_ASIS found as False
VERBOSE2 SINGULARITY_ROOTFS found as /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3
VERBOSE2 SINGULARITY_METADATA_FOLDER found as /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3/.singularity.d
VERBOSE2 SINGULARITY_FIX_PERMS found as False
VERBOSE2 SINGULARITY_COLORIZE not defined (None)
VERBOSE2 SINGULARITY_DISABLE_CACHE found as False
VERBOSE2 SINGULARITY_CACHEDIR found as /home/newuser/.singularity
VERBOSE2 SINGULARITY_ENVBASE found as /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3/.singularity.d/env
VERBOSE2 SINGULARITY_LABELFILE found as /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3/.singularity.d/labels.json
VERBOSE2 SINGULARITY_INCLUDECMD found as False
VERBOSE2 SINGULARITY_PULLFOLDER found as /home/newuser
VERBOSE2 SHUB_NAMEBYHASH not defined (None)
VERBOSE2 SHUB_NAMEBYCOMMIT not defined (None)
VERBOSE2 SHUB_CONTAINERNAME not defined (None)
VERBOSE2 SINGULARITY_CONTENTS found as /tmp/.singularity-layers.p05BbE5N
VERBOSE2 SINGULARITY_PYTHREADS found as 9
VERBOSE2 SINGULARITY_CONTAINER found as docker://bids/mriqc:0.9.3
DEBUG Found uri docker://
DEBUG
*** STARTING DOCKER IMPORT PYTHON  ****
DEBUG Docker layers and (env,labels,runscript) will be written to: /tmp/.singularity-layers.p05BbE5N
VERBOSE2 SINGULARITY_DOCKER_USERNAME not defined (None)
VERBOSE2 SINGULARITY_DOCKER_PASSWORD found
DEBUG Starting Docker IMPORT, includes environment, runscript, and metadata.
VERBOSE Docker image: bids/mriqc:0.9.3
VERBOSE2 Specified Docker ENTRYPOINT as %runscript.
DEBUG Headers found: Content-Type,Accept
VERBOSE Registry: index.docker.io
VERBOSE Namespace: bids
VERBOSE Repo Name: mriqc
VERBOSE Repo Tag: 0.9.3
VERBOSE Version: None
VERBOSE Obtaining tags: https://index.docker.io/v2/bids/mriqc/tags/list
DEBUG GET https://index.docker.io/v2/bids/mriqc/tags/list
DEBUG Http Error with code 401
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Authorization,Accept
Docker image path: index.docker.io/bids/mriqc:0.9.3
VERBOSE Obtaining manifest: https://index.docker.io/v2/bids/mriqc/manifests/0.9.3
DEBUG GET https://index.docker.io/v2/bids/mriqc/manifests/0.9.3
DEBUG Image manifest version 2.1 found.
DEBUG Adding digest sha256:243eedf190b75e91c01f330ff21c77f4c6dc385d7756dda9e7f39741ad4f4071
DEBUG Adding digest sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBUG Adding digest sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83
DEBUG Adding digest sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439
DEBUG Adding digest sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f
DEBUG Adding digest sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac
DEBUG Adding digest sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0
DEBUG Adding digest sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5
DEBUG Adding digest sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d
DEBUG Adding digest sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810
DEBUG Adding digest sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274
DEBUG Adding digest sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb
DEBUG Adding digest sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39
DEBUG Adding digest sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa
DEBUG Adding digest sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351
DEBUG Adding digest sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163
DEBUG Adding digest sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef
DEBUG Adding digest sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d
DEBUG Adding digest sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3
DEBUG Adding digest sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8
DEBUG Adding digest sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6
DEBUG Adding digest sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608
DEBUG Adding digest sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2
DEBUG Adding digest sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714
DEBUG Adding digest sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787
VERBOSE Obtaining manifest: https://index.docker.io/v2/bids/mriqc/manifests/0.9.3
DEBUG GET https://index.docker.io/v2/bids/mriqc/manifests/0.9.3
Cache folder set to /home/newuser/.singularity/docker
DEBUG Using 9 workers for multiprocess.
[1/25] ||----------------------------------|   0.0% DEBUG Starting multiprocess
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:243eedf190b75e91c01f330ff21c77f4c6dc385d7756dda9e7f39741ad4f4071
DEBUG Downloading layer sha256:243eedf190b75e91c01f330ff21c77f4c6dc385d7756dda9e7f39741ad4f4071
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:243eedf190b75e91c01f330ff21c77f4c6dc385d7756dda9e7f39741ad4f4071
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBUG Downloading layer sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83
DEBUG Downloading layer sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439
DEBUG Downloading layer sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f
DEBUG Downloading layer sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac
DEBUG Downloading layer sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0
DEBUG Downloading layer sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5
DEBUG Downloading layer sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d
DEBUG Downloading layer sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:243eedf190b75e91c01f330ff21c77f4c6dc385d7756dda9e7f39741ad4f4071.tar.gz.U20j_t
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz.0gYqfZ
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439.tar.gz.V30bYP
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5.tar.gz.IH41CT
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810
DEBUG Downloading layer sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274
DEBUG Downloading layer sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb
DEBUG Downloading layer sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39
DEBUG Downloading layer sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810.tar.gz.LtQorU
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac.tar.gz.wwvzRR
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa
DEBUG Downloading layer sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39.tar.gz.foZsAE
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351
DEBUG Downloading layer sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163
DEBUG Downloading layer sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0.tar.gz.EZ5I6N
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f.tar.gz.PjJTg_
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef
DEBUG Downloading layer sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d
DEBUG Downloading layer sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163.tar.gz.yADGfK
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3
DEBUG Downloading layer sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351.tar.gz.mRxm_M
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d.tar.gz.vGotsS
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8
DEBUG Downloading layer sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa.tar.gz.1JgtYL
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6
DEBUG Downloading layer sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608
DEBUG Downloading layer sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8.tar.gz.mFAqrR
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6.tar.gz.HxXbJw
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2
DEBUG Downloading layer sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608.tar.gz.Pd9YfA
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714
DEBUG Downloading layer sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714
DEBUG Headers found: Content-Type,Accept,Authorization
VERBOSE Downloading layers from https://index.docker.io/v2/bids/mriqc/blobs/sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787
DEBUG Downloading layer sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787
DEBUG GET (stream) https://index.docker.io/v2/bids/mriqc/blobs/sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2.tar.gz.v0ZLeT
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714.tar.gz.oGuiEV
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
DEBUG Headers found: Content-Type,Accept,Authorization
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb.tar.gz.rI8_yE
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787.tar.gz.WcVXpQ
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
[6/25] |========|--------------------------|  24.0% DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3.tar.gz.rXZoaA
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
[7/25] |=========|-------------------------|  28.0% DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83.tar.gz.Ifc1sU
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d.tar.gz.iXpjl4
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
[8/25] |===========|-----------------------|  32.0% DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef.tar.gz.ZUgBLs
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
[14/25] |===================|---------------|  56.0% DEBUG Download of raw file (pre permissions fix) is /home/newuser/.singularity/docker/sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274.tar.gz.Pna_NH
DEBUG GET https://auth.docker.io/token?service=registry.docker.io&expires_in=9000&scope=repository:bids/mriqc:pull
DEBUG Headers found: Content-Type,Accept,Authorization
[25/25] |===================================| 100.0%
DEBUG Ending multiprocess, runtime: 64.0388839245 sec
VERBOSE3 Found Docker command (Entrypoint) /usr/local/miniconda/bin/mriqc
VERBOSE3 Found Docker command (Cmd) None
VERBOSE3 Adding Docker ENTRYPOINT as Singularity runscript...
DEBUG /usr/local/miniconda/bin/mriqc
VERBOSE3 Found Docker command (Env) PATH=/usr/local/miniconda/bin:/opt/ants:/usr/lib/fsl/5.0:/usr/lib/afni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
FSLDIR=/usr/share/fsl/5.0
FSLOUTPUTTYPE=NIFTI_GZ
FSLMULTIFILEQUIT=TRUE
POSSUMDIR=/usr/share/fsl/5.0
LD_LIBRARY_PATH=/usr/lib/fsl/5.0:
FSLTCLSH=/usr/bin/tclsh
FSLWISH=/usr/bin/wish
AFNI_MODELPATH=/usr/lib/afni/models
AFNI_IMSAVE_WARNINGS=NO
AFNI_TTATLAS_DATASET=/usr/share/afni/atlases
AFNI_PLUGINPATH=/usr/lib/afni/plugins
ANTSPATH=/opt/ants
LANG=C.UTF-8
LC_ALL=C.UTF-8
CRN_SHARED_DATA=/niworkflows_data
VERBOSE3 Found Docker container environment!
VERBOSE3 Adding Docker environment to metadata tar
DEBUG Found template for tarinfo
VERBOSE3 Found Docker command (Labels) {u'org.label-schema.version': u'0.9.3', u'org.label-schema.schema-version': u'1.0', u'org.label-schema.vcs-ref': u'de60ff0', u'org.label-schema.url': u'http://mriqc.readthedocs.io', u'org.label-schema.vcs-url': u'https://github.com/poldracklab/mriqc', u'org.label-schema.build-date': u'2017-04-23T18:10:33Z', u'org.label-schema.name': u'MRIQC', u'org.label-schema.description': u'MRIQC  - NR-IQMs (no-reference Image Quality Metrics) for MRI'}
VERBOSE3 Found Docker container labels!
VERBOSE3 Adding Docker labels to metadata tar
DEBUG Found template for tarinfo
VERBOSE3 Adding Docker runscript to metadata tar
DEBUG Found template for tarinfo
VERBOSE2 Tar file with Docker env and labels: /home/newuser/.singularity/metadata/sha256:2bf1a0cf6f2a1259591399f9afe1956e03128633732c390aa0889af805e1b9a4.tar.gz
VERBOSE3 Writing Docker layers files to /tmp/.singularity-layers.p05BbE5N
VERBOSE2 Writing file /tmp/.singularity-layers.p05BbE5N with mode w.
VERBOSE2 Writing file /tmp/.singularity-layers.p05BbE5N with mode a.
DEBUG *** FINISHING DOCKER IMPORT PYTHON PORTION ****
Creating container runtime...
Importing: base Singularity environment
Exploding layer: sha256:243eedf190b75e91c01f330ff21c77f4c6dc385d7756dda9e7f39741ad4f4071.tar.gz
Exploding layer: sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4.tar.gz
Exploding layer: sha256:a4936036b8465027fd8e6a51cb5cae33190f705b4c4d17a8acf41f471c87ca83.tar.gz
Exploding layer: sha256:d71212883635f51c04e6030dc23dad7a8a5db1a013eafed23fcc976db3ef3439.tar.gz
Exploding layer: sha256:736625e6e08f2b6b983428e79a67a40acfb9647ab17184e9e2e911cd0a0c6f4f.tar.gz
Exploding layer: sha256:de5b15143d20c60c776fd5cd31ca09caf1e293ac50f98f18bcde5dc6a84d55ac.tar.gz
Exploding layer: sha256:54a5564a38ad416b14910a6dc41fec03ff8df40581d9cf026c5f05e4d3ef26f0.tar.gz
Exploding layer: sha256:91f577f6f911091eb01b3f1819860d92c4e1fb8550f37773088f0988fdd14cf5.tar.gz
Exploding layer: sha256:2f506b00f891f2e84f62cf1acdf59423eda26f37be82f7e39e2d5945e55b707d.tar.gz
Exploding layer: sha256:12f2072347221f221d1676db9f33d123fa4540f290446e0ac7722ed4ef0a0810.tar.gz
Exploding layer: sha256:9ddcf08966105c7e4be67e90fa25881617485ab6cfd74072cd3a1ae913301274.tar.gz
Exploding layer: sha256:d4a29c916103cde68802c58bf4ec909216cc4972895d2d5b1d448b0c8b17eceb.tar.gz
Exploding layer: sha256:7f4975bb38892fcdb9285fa22776bdd387c1c947c26aa0c7c3b62a0e8650df39.tar.gz
Exploding layer: sha256:83539119cf7d0c4d2076e8cdde9dcce2547aca339d0f1a4b6e28b28ae6933caa.tar.gz
Exploding layer: sha256:594bf226e62babfe551e6a88838ef0c5798a447ecedd65d5ac69abce4ae2d351.tar.gz
Exploding layer: sha256:399f9aadedac10094e52dc7d685ef8a04e54074bd157ebf1257cc0c4cedf7163.tar.gz
Exploding layer: sha256:7d6bbcadbe0101e0f7deb1b6a83d2a520e805447e5afd48303a0229c034df2ef.tar.gz
Exploding layer: sha256:d9e9382a21dd7dfe6babf8a4e744cafd6f2f7ebb983e2d40c7cb13fbc98b115d.tar.gz
Exploding layer: sha256:6f06d723971b4f12e240f3336a17bf45057b575e501bc65071d4298764d892e3.tar.gz
Exploding layer: sha256:91641bf41cb3485b405e4b2a287d8abc2c5545367faad1e2f72f103bff9c18f8.tar.gz
Exploding layer: sha256:bd842a2105a862f62ba4ad726531029be49046c290307e9c07a688c89f4606e6.tar.gz
Exploding layer: sha256:4a70713c436f2350e8a9f1c1e6c823290f331254c6c21d5657056499d8575608.tar.gz
Exploding layer: sha256:63fe8c0068a811103c702452985368fc99014f83b5f0151cf631644f13f383b2.tar.gz
Exploding layer: sha256:4daad8bdde31f5969d6a44800e993a911bed25853b2f89417b07ce7f0ac67714.tar.gz
Exploding layer: sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787.tar.gz
Exploding layer: sha256:2bf1a0cf6f2a1259591399f9afe1956e03128633732c390aa0889af805e1b9a4.tar.gz
VERBOSE [U=0,P=24]         message_init()                            Set messagelevel to: 5
VERBOSE [U=0,P=24]         singularity_config_parse()                Initialize configuration file: /usr/local/etc/singularity/singularity.conf
DEBUG   [U=0,P=24]         singularity_config_parse()                Starting parse of configuration file /usr/local/etc/singularity/singularity.conf
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key allow setuid = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key max loop devices = '256'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key allow pid ns = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key config passwd = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key config group = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key config resolv_conf = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key mount proc = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key mount sys = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key mount dev = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key mount home = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key mount tmp = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key mount hostfs = 'no'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key bind path = '/etc/hosts'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key user bind control = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key enable overlay = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key mount slave = 'yes'
VERBOSE [U=0,P=24]         singularity_config_parse()                Got config key container dir = '/var/lib/singularity/mnt'
DEBUG   [U=0,P=24]         singularity_config_parse()                Finished parsing configuration file '/usr/local/etc/singularity/singularity.conf'
VERBOSE [U=0,P=24]         singularity_registry_init()               Initializing Singularity Registry
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'LIBEXECDIR' = '/usr/local/libexec'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(libexecdir, /usr/local/libexec) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'COMMAND' = 'exec'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(COMMAND, exec) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'MESSAGELEVEL' = '5'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(MESSAGELEVEL, 5) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'ROOTFS' = '/tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(ROOTFS, /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'VERSION' = '2.2.99-HEAD.g945c6ee'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(version, 2.2.99-HEAD.g945c6ee) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'LOCALSTATEDIR' = '/usr/local/var'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(localstatedir, /usr/local/var) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'CONTENTS' = '/tmp/.singularity-layers.p05BbE5N'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(CONTENTS, /tmp/.singularity-layers.p05BbE5N) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'SYSCONFDIR' = '/usr/local/etc'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(sysconfdir, /usr/local/etc) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'BINDIR' = '/usr/local/bin'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(bindir, /usr/local/bin) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'CONTAINER' = 'docker://bids/mriqc:0.9.3'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(CONTAINER, docker://bids/mriqc:0.9.3) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'NOSESSIONCLEANUP' = '1'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(NOSESSIONCLEANUP, 1) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'IMAGE' = '/tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(IMAGE, /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3) = 0
VERBOSE [U=0,P=24]         singularity_registry_set()                Adding value to registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.8Wthh452'
DEBUG   [U=0,P=24]         singularity_registry_set()                Returning singularity_registry_set(SESSIONDIR, /tmp/.singularity-runtime.8Wthh452) = 0
DEBUG   [U=0,P=24]         singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=0,P=24]         singularity_registry_get()                Returning NULL on 'TARGET_UID'
DEBUG   [U=0,P=24]         singularity_registry_get()                Returning NULL on 'TARGET_GID'
DEBUG   [U=0,P=24]         singularity_priv_init()                   Initializing user info
DEBUG   [U=0,P=24]         singularity_priv_init()                   Set the calling user's username to: newuser
DEBUG   [U=0,P=24]         singularity_priv_init()                   Marking uinfo structure as ready
DEBUG   [U=0,P=24]         singularity_priv_init()                   Obtaining home directory
VERBOSE [U=0,P=24]         singularity_priv_init()                   Set home (via getpwuid()) to: /home/newuser
VERBOSE [U=0,P=24]         singularity_suid_init()                   Running SUID program workflow
VERBOSE [U=0,P=24]         singularity_suid_init()                   Checking program has appropriate permissions
VERBOSE [U=0,P=24]         singularity_suid_init()                   Checking configuration file is properly owned by root
VERBOSE [U=0,P=24]         singularity_suid_init()                   Checking if singularity.conf allows us to run as suid
DEBUG   [U=0,P=24]         singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow setuid, yes)
DEBUG   [U=0,P=24]         singularity_config_get_value_impl()       Returning configuration value allow setuid='yes'
DEBUG   [U=0,P=24]         singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow setuid, yes) = 1
DEBUG   [U=0,P=24]         singularity_registry_get()                Returning NULL on 'NOSUID'
VERBOSE [U=0,P=24]         singularity_priv_userns()                 Invoking the user namespace
DEBUG   [U=0,P=24]         singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow user ns, yes)
DEBUG   [U=0,P=24]         singularity_config_get_value_impl()       No configuration entry found for 'allow user ns'; returning default value 'yes'
DEBUG   [U=0,P=24]         singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow user ns, yes) = 1
VERBOSE [U=0,P=24]         singularity_priv_userns()                 Not virtualizing USER namespace: running as SUID
DEBUG   [U=0,P=24]         singularity_priv_userns()                 Returning singularity_priv_init(void)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.8Wthh452'
DEBUG   [U=500,P=24]       singularity_sessiondir()                  Got SINGULARITY_SESSIONDIR: (null)
VERBOSE [U=500,P=24]       singularity_sessiondir()                  Creating session directory: /tmp/.singularity-runtime.8Wthh452
DEBUG   [U=500,P=24]       singularity_sessiondir()                  Opening sessiondir file descriptor
DEBUG   [U=500,P=24]       singularity_sessiondir()                  Setting shared flock() on session directory
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'NOSESSIONCLEANUP' = '1'
DEBUG   [U=500,P=24]       singularity_sessiondir()                  Not spawning a session directory cleanup process
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'IMAGE' = '/tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3'
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'WRITABLE'
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       No configuration entry found for 'limit container owners'; returning default value 'NULL'
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       No configuration entry found for 'limit container paths'; returning default value 'NULL'
DEBUG   [U=500,P=24]       singularity_image_open()                  Opening file descriptor to image: /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3
DEBUG   [U=500,P=24]       singularity_runtime_ns()                  Calling: _singularity_runtime_ns_pid()
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow pid ns, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value allow pid ns='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow pid ns, yes) = 1
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'UNSHARE_PID'
VERBOSE [U=500,P=24]       singularity_runtime_ns_pid()              Not virtualizing PID namespace on user request
DEBUG   [U=500,P=24]       singularity_runtime_ns()                  Calling: _singularity_runtime_ns_ipc()
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(allow ipc ns, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       No configuration entry found for 'allow ipc ns'; returning default value 'yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(allow ipc ns, yes) = 1
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'UNSHARE_IPC'
VERBOSE [U=500,P=24]       singularity_runtime_ns_ipc()              Not virtualizing IPC namespace on user request
DEBUG   [U=500,P=24]       singularity_runtime_ns()                  Calling: _singularity_runtime_ns_mnt()
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount slave, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount slave='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount slave, yes) = 1
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=24]         singularity_runtime_ns_mnt()              Virtualizing FS namespace
DEBUG   [U=0,P=24]         singularity_runtime_ns_mnt()              Virtualizing mount namespace
DEBUG   [U=0,P=24]         singularity_runtime_ns_mnt()              Making mounts slave
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value max loop devices='256'
DEBUG   [U=500,P=24]       singularity_image_bind()                  Entered singularity_image_bind()
DEBUG   [U=500,P=24]       singularity_image_bind()                  Converting max_loop_devs_string to int: '256'
DEBUG   [U=500,P=24]       singularity_image_bind()                  Converted max_loop_devs_string to int: '256' -> 256
DEBUG   [U=500,P=24]       singularity_image_bind()                  Checking if this image has been properly opened
DEBUG   [U=500,P=24]       singularity_image_bind()                  Checking if image is valid file
VERBOSE [U=500,P=24]       singularity_image_bind()                  Skipping image bind, container is not a file
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value container dir='/var/lib/singularity/mnt'
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Setting container directory to: /var/lib/singularity/mnt/source
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=24]         singularity_runtime_rootfs()              Creating top level source mount directory to: /var/lib/singularity/mnt/source
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity/mnt
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity/mnt/source
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/source
DEBUG   [U=500,P=24]       chk_mode()                                Checking exact mode (40755) on: /var/lib/singularity/mnt/source
DEBUG   [U=500,P=24]       chk_mode()                                Found appropriate mode on file: /var/lib/singularity/mnt/source
VERBOSE [U=500,P=24]       singularity_image_mount()                 Checking what kind of image we are mounting
DEBUG   [U=500,P=24]       singularity_image_mount_squashfs_check()  Image does not appear to be of type '.sqsh': /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3
DEBUG   [U=500,P=24]       singularity_image_mount_dir_check()       Checking if image is a directory
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=24]         singularity_image_mount_dir_mount()       Mounting container directory /tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3->/var/lib/singularity/mnt/source
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/source
DEBUG   [U=500,P=24]       action_ready()                            Checking if container is valid at: /var/lib/singularity/mnt/source
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/source
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value container dir='/var/lib/singularity/mnt'
DEBUG   [U=500,P=24]       singularity_runtime_overlayfs()           Checking if overlayfs should be used
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(enable overlay, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value enable overlay='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(enable overlay, yes) = 1
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'DISABLE_OVERLAYFS'
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'WRITABLE'
DEBUG   [U=500,P=24]       singularity_runtime_overlayfs()           OverlayFS enabled by host build
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=24]         singularity_runtime_overlayfs()           Creating top level overlay mount directory: /var/lib/singularity/mnt/overlay
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay
DEBUG   [U=0,P=24]         singularity_runtime_overlayfs()           Mounting overlay tmpfs: /var/lib/singularity/mnt/overlay
DEBUG   [U=0,P=24]         singularity_runtime_overlayfs()           Creating upper overlay directory: /var/lib/singularity/mnt/overlay/upper
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay/upper
DEBUG   [U=0,P=24]         singularity_runtime_overlayfs()           Creating overlay work directory: /var/lib/singularity/mnt/overlay/work
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity/mnt/overlay/work
DEBUG   [U=0,P=24]         singularity_runtime_overlayfs()           Creating mount_final directory: /var/lib/singularity/mnt/final
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity/mnt/final
VERBOSE [U=0,P=24]         singularity_runtime_overlayfs()           Mounting overlay with options: lowerdir=/var/lib/singularity/mnt/source,upperdir=/var/lib/singularity/mnt/overlay/upper,workdir=/var/lib/singularity/mnt/overlay/work
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
VERBOSE [U=500,P=24]       singularity_registry_set()                Adding value to registry: 'OVERLAYFS_ENABLED' = '1'
DEBUG   [U=500,P=24]       singularity_registry_set()                Returning singularity_registry_set(OVERLAYFS_ENABLED, 1) = 0
VERBOSE [U=500,P=24]       singularity_runtime_overlayfs()           Updating the containerdir to: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Setting container_directory = '/var/lib/singularity/mnt/final'
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
VERBOSE [U=500,P=24]       singularity_runtime_mounts()              Running all mount components
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount hostfs, no)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount hostfs='no'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount hostfs, no) = 0
DEBUG   [U=500,P=24]       singularity_runtime_mount_hostfs()        Not mounting host file systems per configuration
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=500,P=24]       singularity_runtime_mount_binds()         Checking configuration file for 'bind path'
VERBOSE [U=500,P=24]       singularity_runtime_mount_binds()         Found 'bind path' = /etc/hosts, /etc/hosts
DEBUG   [U=500,P=24]       singularity_runtime_mount_binds()         Checking if bind point is already mounted: /etc/hosts
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       check_mounted()                           Opening /proc/mounts
DEBUG   [U=500,P=24]       check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_mount_binds()         Binding '/etc/hosts' to '/var/lib/singularity/mnt/final//etc/hosts'
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount proc'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount proc, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount proc='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount proc, yes) = 1
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'PIDNS_ENABLED'
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_mount_kernelfs()      Bind-mounting host /proc
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_mount_kernelfs()      Checking configuration file for 'mount sys'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount sys, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount sys='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount sys, yes) = 1
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_mount_kernelfs()      Mounting /sys
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
DEBUG   [U=500,P=24]       singularity_runtime_mount_dev()           Checking configuration file for 'mount dev'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount dev, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount dev='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount dev, yes) = 1
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_mount_dev()           Bind mounting /dev
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount tmp, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount tmp='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount tmp, yes) = 1
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_mount_tmp()           Mounting directory: /tmp
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_mount_tmp()           Mounting directory: /var/tmp
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(mount home, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value mount home='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(mount home, yes) = 1
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Checking that home directry is configured: /home/newuser
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Checking if SINGULARITY_CONTAIN is set
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Checking if home directories are being influenced by user
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'HOME'
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Checking ownership of home directory source: /home/newuser
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Checking to make sure home directory destination is a full path: /home/newuser
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Checking if home directory is already mounted: /home/newuser
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       check_mounted()                           Opening /proc/mounts
DEBUG   [U=500,P=24]       check_mounted()                           Iterating through /proc/mounts
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Checking if overlay is enabled
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'OVERLAYFS_ENABLED' = '1'
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Binding home directory direct (no staging)
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Setting home directory source to: '/home/newuser'
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Setting home directory dest to: '/var/lib/singularity/mnt/final' + '/home/newuser'
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Creating home directry within container: /var/lib/singularity/mnt/final/home/newuser
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=24]         s_mkpath()                                Creating directory: /var/lib/singularity/mnt/final/home/newuser
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_mount_home()          Created home directory within the container: /var/lib/singularity/mnt/final/home/newuser
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_mount_home()          Mounting home directory source into container: /home/newuser -> /var/lib/singularity/mnt/final/home/newuser
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       envar_set()                               Setting environment variable: 'HOME' = '/home/newuser'
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_runtime_mount_userbinds()     Checking for environment variable 'SINGULARITY_BINDPATH'
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'BINDPATH'
DEBUG   [U=500,P=24]       singularity_runtime_mount_userbinds()     No user bind mounts specified.
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_runtime_mount_scratch()       Getting SINGULARITY_SCRATCHDIR from environment
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'SCRATCHDIR'
DEBUG   [U=500,P=24]       singularity_runtime_mount_scratch()       Not mounting scratch directory: Not requested
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_runtime_mount_cwd()           Checking to see if we should mount current working directory
DEBUG   [U=500,P=24]       singularity_runtime_mount_cwd()           Getting current working directory
DEBUG   [U=500,P=24]       singularity_runtime_mount_cwd()           Checking if current directory exists in container
DEBUG   [U=500,P=24]       file_devino()                             Called file_devino(/home/newuser)
DEBUG   [U=500,P=24]       file_devino()                             Returning file_devino(/home/newuser) = 59.873741
DEBUG   [U=500,P=24]       file_devino()                             Called file_devino(/var/lib/singularity/mnt/final/home/newuser)
DEBUG   [U=500,P=24]       file_devino()                             Returning file_devino(/var/lib/singularity/mnt/final/home/newuser) = 59.873741
DEBUG   [U=500,P=24]       singularity_runtime_mount_cwd()           Checking if container's cwd == host's cwd
VERBOSE [U=500,P=24]       singularity_runtime_mount_cwd()           Not mounting current directory: location already available within container
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
VERBOSE [U=500,P=24]       singularity_runtime_files()               Running file components
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.8Wthh452'
DEBUG   [U=500,P=24]       singularity_runtime_files_passwd()        Called singularity_file_passwd_create()
DEBUG   [U=500,P=24]       singularity_runtime_files_passwd()        Checking configuration option: 'config passwd'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config passwd, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value config passwd='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config passwd, yes) = 1
VERBOSE [U=500,P=24]       singularity_runtime_files_passwd()        Checking for template passwd file: /var/lib/singularity/mnt/final/etc/passwd
VERBOSE [U=500,P=24]       singularity_runtime_files_passwd()        Creating template of /etc/passwd
DEBUG   [U=500,P=24]       copy_file()                               Called copy_file(/var/lib/singularity/mnt/final/etc/passwd, /tmp/.singularity-runtime.8Wthh452/passwd)
DEBUG   [U=500,P=24]       copy_file()                               Opening source file: /var/lib/singularity/mnt/final/etc/passwd
DEBUG   [U=500,P=24]       copy_file()                               Opening destination file: /tmp/.singularity-runtime.8Wthh452/passwd
DEBUG   [U=500,P=24]       copy_file()                               Calling fstat() on source file descriptor: 5
DEBUG   [U=500,P=24]       copy_file()                               Cloning permission string of source to dest
DEBUG   [U=500,P=24]       copy_file()                               Copying file data...
DEBUG   [U=500,P=24]       copy_file()                               Done copying data, closing file pointers
DEBUG   [U=500,P=24]       copy_file()                               Returning copy_file(/var/lib/singularity/mnt/final/etc/passwd, /tmp/.singularity-runtime.8Wthh452/passwd) = 0
VERBOSE [U=500,P=24]       singularity_runtime_files_passwd()        Creating template passwd file and appending user data: /tmp/.singularity-runtime.8Wthh452/passwd
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       container_file_bind()                     Called file_bind(/tmp/.singularity-runtime.8Wthh452/passwd, /etc/passwd()
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         container_file_bind()                     Binding file '/tmp/.singularity-runtime.8Wthh452/passwd' to '/var/lib/singularity/mnt/final/etc/passwd'
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       envar_set()                               Setting environment variable: 'HOME' = '/home/newuser'
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.8Wthh452'
DEBUG   [U=500,P=24]       singularity_runtime_files_group()         Called singularity_file_group_create()
DEBUG   [U=500,P=24]       singularity_runtime_files_group()         Checking configuration option: 'config group'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config group, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value config group='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config group, yes) = 1
VERBOSE [U=500,P=24]       singularity_runtime_files_group()         Creating template of /etc/group for containment
DEBUG   [U=500,P=24]       copy_file()                               Called copy_file(/var/lib/singularity/mnt/final/etc/group, /tmp/.singularity-runtime.8Wthh452/group)
DEBUG   [U=500,P=24]       copy_file()                               Opening source file: /var/lib/singularity/mnt/final/etc/group
DEBUG   [U=500,P=24]       copy_file()                               Opening destination file: /tmp/.singularity-runtime.8Wthh452/group
DEBUG   [U=500,P=24]       copy_file()                               Calling fstat() on source file descriptor: 5
DEBUG   [U=500,P=24]       copy_file()                               Cloning permission string of source to dest
DEBUG   [U=500,P=24]       copy_file()                               Copying file data...
DEBUG   [U=500,P=24]       copy_file()                               Done copying data, closing file pointers
DEBUG   [U=500,P=24]       copy_file()                               Returning copy_file(/var/lib/singularity/mnt/final/etc/group, /tmp/.singularity-runtime.8Wthh452/group) = 0
VERBOSE [U=500,P=24]       singularity_runtime_files_group()         Updating group file with user info
DEBUG   [U=500,P=24]       singularity_runtime_files_group()         Getting supplementary group info
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       container_file_bind()                     Called file_bind(/tmp/.singularity-runtime.8Wthh452/group, /etc/group()
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         container_file_bind()                     Binding file '/tmp/.singularity-runtime.8Wthh452/group' to '/var/lib/singularity/mnt/final/etc/group'
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_files_resolvconf()    Checking configuration option
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Called singularity_config_get_bool(config resolv_conf, yes)
DEBUG   [U=500,P=24]       singularity_config_get_value_impl()       Returning configuration value config resolv_conf='yes'
DEBUG   [U=500,P=24]       singularity_config_get_bool_char_impl()   Return singularity_config_get_bool(config resolv_conf, yes) = 1
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       container_file_bind()                     Called file_bind(/etc/resolv.conf, /etc/resolv.conf()
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         container_file_bind()                     Binding file '/etc/resolv.conf' to '/var/lib/singularity/mnt/final/etc/resolv.conf'
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'SESSIONDIR' = '/tmp/.singularity-runtime.8Wthh452'
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'CONTAINLIBS'
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
VERBOSE [U=500,P=24]       singularity_runtime_enter()               Containing all rootfs components
DEBUG   [U=500,P=24]       singularity_runtime_rootfs()              Returning container_directory: /var/lib/singularity/mnt/final
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
VERBOSE [U=0,P=24]         singularity_runtime_enter_chroot()        Entering container file system root: /var/lib/singularity/mnt/final
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Dropping privileges to UID=500, GID=500 (0 supplementary GIDs)
DEBUG   [U=0,P=24]         singularity_priv_drop()                   Restoring supplementary groups
DEBUG   [U=500,P=24]       singularity_priv_drop()                   Confirming we have correct UID/GID
DEBUG   [U=500,P=24]       singularity_runtime_enter_chroot()        Changing dir to '/' within the new root
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Cloning environment
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Counted 25 environment elements
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'CLEANENV'
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Cleaning environment
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: HOSTNAME=0e726194b97a
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: TERM=xterm
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_libexecdir=/usr/local/libexec
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_libexecdir
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_COMMAND=exec
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_COMMAND
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_MESSAGELEVEL=5
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_MESSAGELEVEL
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: USER=newuser
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;(TRUNCATED...)
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_ROOTFS=/tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_ROOTFS
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_version=2.2.99-HEAD.g945c6ee
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_version
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_localstatedir=/usr/local/var
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_localstatedir
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: PWD=/home/newuser
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: LANG=en_US.UTF-8
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SHLVL=1
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: HOME=/home/newuser
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CONTENTS=/tmp/.singularity-layers.p05BbE5N
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTENTS
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_sysconfdir=/usr/local/etc
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_sysconfdir
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_bindir=/usr/local/bin
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_bindir
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: LESSOPEN=||/usr/bin/lesspipe.sh %s
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_CONTAINER=docker://bids/mriqc:0.9.3
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_CONTAINER
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_NOSESSIONCLEANUP=1
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_NOSESSIONCLEANUP
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_IMAGE=/tmp/.singularity-runtime.8Wthh452/container/bids/mriqc:0.9.3
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_IMAGE
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: SINGULARITY_SESSIONDIR=/tmp/.singularity-runtime.8Wthh452
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Unsetting environment variable: SINGULARITY_SESSIONDIR
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: HISTFILE=
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Evaluating envar to clean: G_BROKEN_FILENAMES=1
DEBUG   [U=500,P=24]       singularity_runtime_environment()         Transposing environment
DEBUG   [U=500,P=24]       singularity_priv_drop_perm()              Called singularity_priv_drop_perm(void)
DEBUG   [U=500,P=24]       singularity_priv_drop_perm()              Escalating permissison so we can properly drop permission
DEBUG   [U=500,P=24]       singularity_priv_escalate()               Temporarily escalating privileges (U=500)
DEBUG   [U=0,P=24]         singularity_priv_escalate()               Clearing supplementary GIDs.
DEBUG   [U=0,P=24]         singularity_priv_drop_perm()              Resetting supplementary groups
DEBUG   [U=0,P=24]         singularity_priv_drop_perm()              Dropping to group ID '500'
DEBUG   [U=0,P=24]         singularity_priv_drop_perm()              Dropping real and effective privileges to GID = '500'
DEBUG   [U=0,P=24]         singularity_priv_drop_perm()              Dropping real and effective privileges to UID = '500'
DEBUG   [U=500,P=24]       singularity_priv_drop_perm()              Confirming we have correct GID
DEBUG   [U=500,P=24]       singularity_priv_drop_perm()              Confirming we have correct UID
DEBUG   [U=500,P=24]       singularity_priv_drop_perm()              Setting NO_NEW_PRIVS to prevent future privilege escalations.
DEBUG   [U=500,P=24]       singularity_priv_drop_perm()              Finished dropping privileges
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'CONTAIN'
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'TARGET_PWD'
DEBUG   [U=500,P=24]       main()                                    Attempting to chdir to CWD: /home/newuser
DEBUG   [U=500,P=24]       envar_set()                               Setting environment variable: 'SINGULARITY_CONTAINER' = 'mriqc:0.9.3'
DEBUG   [U=500,P=24]       envar_set()                               Setting environment variable: 'SINGULARITY_NAME' = 'mriqc:0.9.3'
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning NULL on 'SHELL'
VERBOSE [U=500,P=24]       envar_set()                               Not setting envar, null value: SINGULARITY_SHELL
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'COMMAND' = 'exec'
DEBUG   [U=500,P=24]       singularity_registry_get()                Returning value from registry: 'COMMAND' = 'exec'
Singularity: action-suid (U=500,P=24)> USER=newuser, IMAGE='mriqc:0.9.3', COMMAND='exec'

LOG     [U=500,P=24]       main()                                    USER=newuser, IMAGE='mriqc:0.9.3', COMMAND='exec'
VERBOSE [U=500,P=24]       action_exec()                             Exec'ing /.exec
LESSOPEN=||/usr/bin/lesspipe.sh %s
POSSUMDIR=/usr/share/fsl/5.0
AFNI_TTATLAS_DATASET=/usr/share/afni/atlases
AFNI_PLUGINPATH=/usr/lib/afni/plugins
USER=newuser
HOSTNAME=0e726194b97a
SHLVL=1
LD_LIBRARY_PATH=/usr/lib/fsl/5.0::/.singularity.d/libs
HOME=/home/newuser
PS1=Singularity>
G_BROKEN_FILENAMES=1
FSLDIR=/usr/share/fsl/5.0
AFNI_IMSAVE_WARNINGS=NO
ANTSPATH=/opt/ants
FSLMULTIFILEQUIT=TRUE
FSLTCLSH=/usr/bin/tclsh
TERM=xterm
PATH=/usr/local/miniconda/bin:/opt/ants:/usr/lib/fsl/5.0:/usr/lib/afni/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
FSLOUTPUTTYPE=NIFTI_GZ
LANG=C.UTF-8
AFNI_MODELPATH=/usr/lib/afni/models
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
FSLWISH=/usr/bin/wish
CRN_SHARED_DATA=/niworkflows_data
SINGULARITY_CONTAINER=mriqc:0.9.3
PWD=/home/newuser
LC_ALL=C.UTF-8
HISTFILE=
SINGULARITY_NAME=mriqc:0.9.3

Here's the Dockerfile I used:

FROM centos:6

RUN yum -y install git

RUN git clone https://github.com/singularityware/singularity.git && \
    cd singularity && \
    git checkout 945c6ee343a1e6101e22396a90dfdb5944f442b6

RUN yum -y install libtool

RUN cd singularity && \
    ./autogen.sh && \
    ./configure --prefix=/usr/local && \
    make && \
    make install

RUN useradd -ms /bin/bash newuser

USER newuser
WORKDIR /home/newuser
vsoch commented 7 years ago

you are running this with privileged mode? Eg:

docker run -it --privileged vanessa/test

So - I just tried this, with a basic ubuntu image with the Dockerfile and command above, and my entire system crashed. I'm not sure I want to try that again, lol. Is there someone with non ubuntu (mac) that could give this a test? @gmkurtzer ? Here is what you do:

  1. make a folder somewhere, call it test
  2. Copy the file above that chris provided into a file called Dockerfile in that folder
  3. Do docker build -t dockeruser/test .
  4. Then run the image with privileged like
docker run -it --privileged dockeruser/test

then in the image, try something basic first:

singularity run docker://ubuntu

and hold your breath for a possible crash, if it doesn't crash and works ok, try @chrisfilo image

singularity --debug run docker://bids/mriqc:0.9.3

and then see if you get the same error. I think @chrisfilo is either still on Mac or Windows again? There is something funky going on likely because we are using priviledged docker --> docker import via singularity <-- and back.

chrisgorgo commented 7 years ago

Yes I'm using --privileged mode on Windows.

On Tue, May 9, 2017 at 2:25 PM, Vanessa Sochat notifications@github.com wrote:

you are running this with privileged mode? Eg:

docker run -it --privileged vanessa/test

So - I just tried this, with a basic ubuntu image with the Dockerfile and command above, and my entire system crashed. I'm not sure I want to try that again, lol. Is there someone with non ubuntu (mac) that could give this a test? @gmkurtzer https://github.com/gmkurtzer ? Here is what you do:

  1. make a folder somewhere, call it test
  2. Copy the file above that chris provided into a file called Dockerfile in that folder
  3. Do docker build -t dockeruser/test .
  4. Then run the image with privileged like

docker run -it --privileged dockeruser/test

then in the image, try something basic first:

singularity run docker://ubuntu

and hold your breath for a possible crash, if it doesn't crash and works ok, try @chrisfilo https://github.com/chrisfilo image

singularity --debug run docker://bids/mriqc:0.9.3

and then see if you get the same error. I think @chrisfilo https://github.com/chrisfilo is either still on Mac or Windows again? There is something funky going on likely because we are using priviledged docker --> docker import via singularity <-- and back.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/singularityware/singularity/issues/662#issuecomment-300306324, or mute the thread https://github.com/notifications/unsubscribe-auth/AAOkpz2upLrQ39_VbTdJz55DEOhOVQXfks5r4NnbgaJpZM4NUX3f .

gmkurtzer commented 7 years ago

@chrisfilo,

When you ran it earlier there was an error output like:

Failed removing file: /tmp/.singularity-runtime.e9QOcU07/container/bids/mriqc:0.9.3/libwebp-0.5.2-linux-x86-64/include/webp

Can you rerun it again, and if you get that error can you send me the ls -l output of that file please?

Thank you!

vsoch commented 7 years ago

I was about to comment! @gmkurtzer is doing some changes to handle these temporary directories, so another test would be good :)

chrisgorgo commented 7 years ago

I just tested with the latest commit and this issue does not show up anymore. Thanks for the fix @gmkurtzer!

gmkurtzer commented 7 years ago

@chrisfilo I apologize, I realized I broke something else after I committed what you tested. Would you mind running the test one more time please?

Thank you!

chrisgorgo commented 7 years ago

I've tried with 54694d32ba911e1a4e677f15ee3e631217884e64 unfortunately the bug regressed:

[newuser@35853316e77a ~]$ singularity run docker://bids/mriqc:0.9.3
Docker image path: index.docker.io/bids/mriqc:0.9.3
Cache folder set to /home/newuser/.singularity/docker
[25/25] |===================================| 100.0%
Creating container runtime...
Singularity: action-suid (U=500,P=19)> USER=newuser, IMAGE='mriqc:0.9.3', COMMAND='run'

Fontconfig warning: ignoring C.UTF-8: not a valid language tag
usage: mriqc [-h] [-v]
             [--participant_label [PARTICIPANT_LABEL [PARTICIPANT_LABEL ...]]]
             [-m [{T1w,bold,T2w} [{T1w,bold,T2w} ...]]] [-s SESSION_ID]
             [-r RUN_ID] [--nthreads NTHREADS] [--n_procs N_PROCS]
             [--mem_gb MEM_GB] [--write-graph] [--dry-run]
             [--use-plugin USE_PLUGIN] [--ica] [--testing] [--profile]
             [--hmc-afni] [--hmc-fsl] [-f] [--fft-spikes-detector]
             [-w WORK_DIR] [--report-dir REPORT_DIR] [--verbose-reports]
             [--ants-nthreads ANTS_NTHREADS] [--ants-settings ANTS_SETTINGS]
             [--deoblique] [--despike] [--start-idx START_IDX]
             [--stop-idx STOP_IDX] [--correct-slice-timing]
             bids_dir output_dir {participant,group} [{participant,group} ...]
mriqc: error: the following arguments are required: bids_dir, output_dir, analysis_level
[newuser@35853316e77a ~]$ Singularity: Singularity: cleanup (U=500,P=100482)> Failed removing file: /tmp/.singularity-runtime.ZGDDenKs/bids/mriqc:0.9.3/libwebp-0.5.2-linux-x86-64/include/webp

Singularity: Singularity: cleanup (U=500,P=100482)> Could not remove directory /tmp/.singularity-runtime.ZGDDenKs: Directory not empty

Singularity: Singularity: cleanup (U=500,P=100482)> Retval = 255

[newuser@35853316e77a ~]$ ls -al /tmp/.singularity-runtime.ZGDDenKs/bids/mriqc:0.9.3/libwebp-0.5.2-linux-x86-64/include/webp
total 108
drwx------ 2 newuser newuser  4096 Dec 16 00:10 .
drwx------ 3 newuser newuser  4096 Dec 16 00:10 ..
-rw-r--r-- 1 newuser newuser 23112 Dec 16 00:10 decode.h
-rw-r--r-- 1 newuser newuser 15235 Dec 16 00:10 demux.h
-rw-r--r-- 1 newuser newuser 26278 Dec 16 00:10 encode.h
-rw-r--r-- 1 newuser newuser 22885 Dec 16 00:10 mux.h
-rw-r--r-- 1 newuser newuser  3149 Dec 16 00:10 mux_types.h
-rw-r--r-- 1 newuser newuser  1687 Dec 16 00:10 types.h
gmkurtzer commented 7 years ago

Does this program fork or run sub-threads? I think what is happening is that the foreground process returns, but there is still a thread running that hasn't been properly reaped or waited on which is still in that directory.

If you use an image, this shouldn't be an issue anymore, but I don't think I can solve this within singularity.

The reason it seemed to go away before was because in the previous commit I broke the cleanup thread.

chrisgorgo commented 7 years ago

@gmkurtzer I don't think this is related to forking or sub-threading. I get the same behaviour with the ubuntu image and bash:

[newuser@f63a1d2ec534 ~]$ singularity run docker://ubuntu
Docker image path: index.docker.io/library/ubuntu:latest
Cache folder set to /home/newuser/.singularity/docker
[5/5] |===================================| 100.0%
Creating container runtime...
Singularity: action-suid (U=500,P=19)> USER=newuser, IMAGE='ubuntu', COMMAND='run'

newuser@f63a1d2ec534:~$ ls
newuser@f63a1d2ec534:~$ ^C
newuser@f63a1d2ec534:~$ exit
[newuser@f63a1d2ec534 ~]$ Singularity: Singularity: cleanup (U=500,P=5276)> Failed removing file: /tmp/.singularity-runtime.OYoSarbj/ubuntu/root

Singularity: Singularity: cleanup (U=500,P=5276)> Could not remove directory /tmp/.singularity-runtime.OYoSarbj: Directory not empty

Singularity: Singularity: cleanup (U=500,P=5276)> Retval = 255

[newuser@f63a1d2ec534 ~]$ ls -al /tmp/.singularity-runtime.OYoSarbj/ubuntu/root
total 16
drwx------  2 newuser newuser 4096 May 10 16:23 .
drwx------ 21 newuser newuser 4096 May 19 22:59 ..
-rw-r--r--  1 newuser newuser 3106 Oct 22  2015 .bashrc
-rw-r--r--  1 newuser newuser  148 Aug 17  2015 .profile
gmkurtzer commented 7 years ago

That is very weird... I don't get the error ever. :/

vsoch commented 7 years ago

yeah so I (very stupidly) just tried this again - my computer crashed again! At least that's reliable. I'm afraid I'm unable to help, other than to strongly encourage you to come back from the dark side, @chrisfilo :)

dtrudg commented 6 years ago

Closing this, as we believe it's fixed with a PR merged to correct a race condition on some containers, preventing full cleanup. Please let us know if it's still an issue in current Singularity. Thanks!