initial EC2 initialization and installation of datalad and datalad-container
(git)lena:~/proj/repronim/reproman[master]git
$> reproman -l debug create -t aws-ec2 my-nitrc-quick -b instance_type=t2.medium -b image=ami-0acbd99fe8c84efbb
You did not specify an EC2 SSH key-pair name to use when creating your EC2
environment.
Please enter a unique name to create a new key-pair or press [enter] to exit: my-nitrc-quick
That key name exists already, try again: my-nitrc-quick3
2020-12-15 08:35:02,531 [INFO ] Created private key file /home/yoh/.local/share/reproman/ec2_keys/my-nitrc-quick3.pem
2020-12-15 08:35:05,205 [INFO ] Waiting for EC2 instance i-04cc7bfb6534fe133 to start running...
2020-12-15 08:36:06,037 [INFO ] EC2 instance i-04cc7bfb6534fe133 is running!
2020-12-15 08:36:06,037 [INFO ] Waiting for EC2 instance i-04cc7bfb6534fe133 to complete initialization...
2020-12-15 08:38:52,706 [INFO ] EC2 instance i-04cc7bfb6534fe133 initialized!
2020-12-15 08:38:52,729 [INFO ] Created the environment my-nitrc-quick
$> reproman login my-nitrc-quick
ubuntu@nitrcce:~$ sudo apt-get update -qqq
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 InRelease: The following signatures were invalid: EXPKEYSIG 6ED91CA3AC1160CD NVIDIA CORPORATION (Open Source Projects)
W: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64/InRelease The following signatures were invalid: EXPKEYSIG 6ED91CA3AC1160CD NVIDIA CORPORATION (Open Source Projects)
W: Some index files failed to download. They have been ignored, or old ones used instead.
ubuntu@nitrcce:~$ sudo apt-get install -y datalad datalad-container
Reading package lists... Done
...
Get:7 http://neuro.debian.net/debian bionic/main amd64 datalad all 0.13.6-1~nd18.04+1 [137 kB]
Get:8 http://neuro.debian.net/debian bionic/main amd64 python3-datalad all 0.13.6-1~nd18.04+1 [1109 kB]
Get:9 http://neuro.debian.net/debian bionic/main amd64 datalad-container all 1.1.0-1~nd18.04+1 [26.2 kB]
Get:10 http://neuro.debian.net/debian bionic/main amd64 singularity-container amd64 2.6.1-2+nd2~nd18.04+1 [351 kB]
.... took awhile -- that instance is slow :-/ ...
configuration and failed execution
ubuntu@nitrcce:~$ datalad install ///repronim/containers
It is highly recommended to configure Git before using DataLad. Set both 'user.name' and 'user.email' configuration variables.
It is highly recommended to configure Git before using DataLad. Set both 'user.name' and 'user.email' configuration variables.
[INFO ] *** Please tell me who you are.
|
| Run
|
| git config --global user.email "you@example.com"
| git config --global user.name "Your Name"
|
| to set your account's default identity.
| Omit --global to set the identity only in this repository.
|
| fatal: unable to auto-detect email address (got 'ubuntu@nitrcce.(none)')
install(ok): /home/ubuntu/containers (dataset)
ubuntu@nitrcce:~$ git config --global user.email "nobody@example.com"
ubuntu@nitrcce:~$ git config --global user.name "Your Name"
ubuntu@nitrcce:~$ cd containers/
ubuntu@nitrcce:~/containers$ bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')
...
> datalad containers-run -n containers/bids-mriqc --input sourcedata --output . '{inputs}' '{outputs}' participant group
[INFO ] Making sure inputs are available (this may take some time)
get(ok): sourcedata/sub-02/anat/sub-02_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/anat/sub-02_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-02/func/sub-02_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_T1w.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/anat/sub-13_inplaneT2.nii.gz (file) [from s3-PUBLIC...]
get(ok): sourcedata/sub-13/func/sub-13_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...]
get(ok): containers/images/bids/bids-mriqc--0.15.1.sing (file) [from origin...]
[INFO ] == Command start (output follows) =====
2020-12-15 15:35:23,242 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata.
* Output folder: /tmp/repro-3vNHTmV/ds000003-qc.
* Analysis levels: participant, group.
Process Process-2:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 400, in init_mriqc
ignore=['derivatives', 'sourcedata', r'^\..*'])
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 185, in __init__
self._validate_root()
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 318, in _validate_root
raise ValueError("BIDS root does not exist: %s" % self.root)
ValueError: BIDS root does not exist: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata
[INFO ] == Command exit (modification check follows) =====
[INFO ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -d . -r -F .git/COMMIT_EDITMSG'
CommandError: 'containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata' '' participant group' failed with exitcode 1 under /tmp/repro-3vNHTmV/ds000003-qc
more specific reproducing + not-reproducing with direct invocation when already in the container
```shell
ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ singularity exec -W /tmp/singtmp.PacrEs -B /tmp/singtmp.PacrEs/tmp:/tmp -B /tmp/singtmp.PacrEs/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing bash
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ export | grep SING
declare -x SINGULARITY_APPNAME=""
declare -x SINGULARITY_CONTAINER="/tmp/repro-3vNHTmV/ds000003-qc/containers/images/bids/bids-mriqc--0.15.1.sing"
declare -x SINGULARITY_NAME="bids-mriqc--0.15.1.sing"
bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ ls /tmp/repro-3vNHTmV/ds000003-qc/sourcedata
CHANGES README dataset_description.json participants.tsv sub-02 sub-13 task-rhymejudgment_bold.json
bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ /.singularity.d/runscript sourcedata '' participant group
2020-12-15 15:40:41,456 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata.
* Output folder: /tmp/repro-3vNHTmV/ds000003-qc.
* Analysis levels: participant, group.
201215-15:40:47,656 nipype.workflow INFO:
Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
2020-12-15 15:40:47,656 nipype.workflow:INFO Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
^CException ignored in: >
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/sqlalchemy/orm/state.py", line 399, in _cleanup
del self._instance_dict
KeyboardInterrupt
Traceback (most recent call last):
File "/usr/local/miniconda/bin/mriqc", line 10, in
sys.exit(main())
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 221, in main
plugin_settings = retval['plugin_settings']
File "", line 2, in __getitem__
File "/usr/local/miniconda/lib/python3.7/multiprocessing/managers.py", line 795, in _callmethod
conn.send((self._id, methodname, args, kwds))
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 404, in _send_bytes
self._send(header + buf)
File "/usr/local/miniconda/lib/python3.7/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
```
I Ctrl-C'ed above run since it did start going
so it feels like bind mount of /tmp/repro-3vNHTmV/ds000003-qc did not happen... but reason seems to be somewhere due to our singularity_cmd halper since it fails with it:
it fails with it
```shell
ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ bash -x containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata' . participant group
+ set -eu
++ readlink -f containers/scripts/singularity_cmd
+ thisfile=/tmp/repro-3vNHTmV/ds000003-qc/containers/scripts/singularity_cmd
++ dirname /tmp/repro-3vNHTmV/ds000003-qc/containers/scripts/singularity_cmd
+ thisdir=/tmp/repro-3vNHTmV/ds000003-qc/containers/scripts
++ dirname /tmp/repro-3vNHTmV/ds000003-qc/containers/scripts
+ updir=/tmp/repro-3vNHTmV/ds000003-qc/containers
+ BHOME=/tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME
+ cmd=run
+ shift
+ '[' -n '' ']'
++ mktemp -d -t singtmp.XXXXXX
+ tmpdir=/tmp/singtmp.ywiVAe
+ info 'created temp dir /tmp/singtmp.ywiVAe'
+ :
+ trap 'rm -fr "$tmpdir" && info "removed temp dir $tmpdir"' exit
+ pass_git_config user.name 'ReproNim User'
+ var=user.name
+ default='ReproNim User'
++ git config user.name
+ value='Your Name'
+ for attempt in {1..5}
+ git config -f /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME/.gitconfig user.name
+ break
+ pass_git_config user.email nobody@example.com
+ var=user.email
+ default=nobody@example.com
++ git config user.email
+ value=nobody@example.com
+ for attempt in {1..5}
+ git config -f /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME/.gitconfig user.email
+ break
+ SARGS=(-e -B "$PWD" -H "$BHOME" --pwd "$PWD" "$@")
+ need_no_c=
+ for d in "$PWD" "$updir"
+ '[' repro-3vNHTmV/ds000003-qc '!=' /tmp/repro-3vNHTmV/ds000003-qc ']'
+ info 'Creating /tmp/repro-3vNHTmV/ds000003-qc under /tmp/singtmp.ywiVAe'
+ :
+ mkdir -p /tmp/singtmp.ywiVAe//tmp/repro-3vNHTmV/ds000003-qc /tmp/singtmp.ywiVAe/var/tmp
+ need_no_c=1
+ for d in "$PWD" "$updir"
+ '[' repro-3vNHTmV/ds000003-qc/containers '!=' /tmp/repro-3vNHTmV/ds000003-qc/containers ']'
+ info 'Creating /tmp/repro-3vNHTmV/ds000003-qc/containers under /tmp/singtmp.ywiVAe'
+ :
+ mkdir -p /tmp/singtmp.ywiVAe//tmp/repro-3vNHTmV/ds000003-qc/containers /tmp/singtmp.ywiVAe/var/tmp
+ need_no_c=1
+ '[' -z 1 ']'
+ SARGS=(-B "$tmpdir/tmp:/tmp" -B "$tmpdir/var/tmp:/var/tmp" "${SARGS[@]}")
+ hash singularity
+ '[' -z '' ']'
+ singularity run -W /tmp/singtmp.ywiVAe -B /tmp/singtmp.ywiVAe/tmp:/tmp -B /tmp/singtmp.ywiVAe/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing sourcedata . participant group
2020-12-15 15:46:55,399 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata.
* Output folder: /tmp/repro-3vNHTmV/ds000003-qc.
* Analysis levels: group, participant.
Process Process-2:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 400, in init_mriqc
ignore=['derivatives', 'sourcedata', r'^\..*'])
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 185, in __init__
self._validate_root()
File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 318, in _validate_root
raise ValueError("BIDS root does not exist: %s" % self.root)
ValueError: BIDS root does not exist: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata
+ rm -fr /tmp/singtmp.ywiVAe
+ info 'removed temp dir /tmp/singtmp.ywiVAe'
+ :
```
but works if I just go with the direct singularity command it runs:
```shell
ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ singularity run -W /tmp/singtmp.ywiVAe -B /tmp/singtmp.ywiVAe/tmp:/tmp -B /tmp/singtmp.ywiVAe/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing sourcedata '' participant group
2020-12-15 15:47:50,992 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata.
* Output folder: /tmp/repro-3vNHTmV/ds000003-qc.
* Analysis levels: participant, group.
201215-15:47:57,125 nipype.workflow INFO:
Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
2020-12-15 15:47:57,125 nipype.workflow:INFO Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
^C
```
so the reason is really is having that /tmp/singtmp.ywiVAe (and subdirs) which bind mount first and then elderly singularity refuses to bind mount /tmp/repro-3vNHTmV/ds000003-qc on top of it...
quick&dirty workaround is to place TMPDIR somewhere else but /tmp, e.g. mkdir -p ~/tmp and provide it while running the example, e.g. TMPDIR=~/tmp bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')
then we do not bother to create/bind mount those additional paths
```shell
ubuntu@nitrcce:~/tmp/repro-wgnyF4J/ds000003-qc$ TMPDIR=~/tmp bash -x containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata' . participant group
+ set -eu
++ readlink -f containers/scripts/singularity_cmd
+ thisfile=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts/singularity_cmd
++ dirname /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts/singularity_cmd
+ thisdir=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts
++ dirname /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts
+ updir=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers
+ BHOME=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME
+ cmd=run
+ shift
+ '[' -n '' ']'
++ mktemp -d -t singtmp.XXXXXX
+ tmpdir=/home/ubuntu/tmp/singtmp.yq0pJg
+ info 'created temp dir /home/ubuntu/tmp/singtmp.yq0pJg'
+ :
+ trap 'rm -fr "$tmpdir" && info "removed temp dir $tmpdir"' exit
+ pass_git_config user.name 'ReproNim User'
+ var=user.name
+ default='ReproNim User'
++ git config user.name
+ value='Your Name'
+ for attempt in {1..5}
+ git config -f /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME/.gitconfig user.name
+ break
+ pass_git_config user.email nobody@example.com
+ var=user.email
+ default=nobody@example.com
++ git config user.email
+ value=nobody@example.com
+ for attempt in {1..5}
+ git config -f /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME/.gitconfig user.email
+ break
+ SARGS=(-e -B "$PWD" -H "$BHOME" --pwd "$PWD" "$@")
+ need_no_c=
+ for d in "$PWD" "$updir"
+ '[' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc '!=' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc ']'
+ for d in "$PWD" "$updir"
+ '[' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers '!=' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers ']'
+ '[' -z '' ']'
+ SARGS=(-c "${SARGS[@]}")
+ hash singularity
+ '[' -z '' ']'
+ singularity run -W /home/ubuntu/tmp/singtmp.yq0pJg -c -e -B /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc -H /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME --pwd /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing sourcedata . participant group
2020-12-15 16:24:57,803 mriqc:IMPORTANT
Running MRIQC version 0.15.1:
* BIDS dataset path: /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/sourcedata.
* Output folder: /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc.
* Analysis levels: participant, group.
201215-16:25:04,31 nipype.workflow INFO:
Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
2020-12-15 16:25:04,031 nipype.workflow:INFO Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz']
...
```
initial EC2 initialization and installation of datalad and datalad-container
(git)lena:~/proj/repronim/reproman[master]git $> reproman -l debug create -t aws-ec2 my-nitrc-quick -b instance_type=t2.medium -b image=ami-0acbd99fe8c84efbb You did not specify an EC2 SSH key-pair name to use when creating your EC2 environment. Please enter a unique name to create a new key-pair or press [enter] to exit: my-nitrc-quick That key name exists already, try again: my-nitrc-quick3 2020-12-15 08:35:02,531 [INFO ] Created private key file /home/yoh/.local/share/reproman/ec2_keys/my-nitrc-quick3.pem 2020-12-15 08:35:05,205 [INFO ] Waiting for EC2 instance i-04cc7bfb6534fe133 to start running... 2020-12-15 08:36:06,037 [INFO ] EC2 instance i-04cc7bfb6534fe133 is running! 2020-12-15 08:36:06,037 [INFO ] Waiting for EC2 instance i-04cc7bfb6534fe133 to complete initialization... 2020-12-15 08:38:52,706 [INFO ] EC2 instance i-04cc7bfb6534fe133 initialized! 2020-12-15 08:38:52,729 [INFO ] Created the environment my-nitrc-quick $> reproman login my-nitrc-quick ubuntu@nitrcce:~$ sudo apt-get update -qqq W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://nvidia.github.io/nvidia-docker/ubuntu18.04/amd64 InRelease: The following signatures were invalid: EXPKEYSIG 6ED91CA3AC1160CD NVIDIA CORPORATION (Open Source Projects)configuration and failed execution
ubuntu@nitrcce:~$ datalad install ///repronim/containers It is highly recommended to configure Git before using DataLad. Set both 'user.name' and 'user.email' configuration variables. It is highly recommended to configure Git before using DataLad. Set both 'user.name' and 'user.email' configuration variables. [INFO ] *** Please tell me who you are. | | Run | | git config --global user.email "you@example.com" | git config --global user.name "Your Name" | | to set your account's default identity. | Omit --global to set the identity only in this repository. | | fatal: unable to auto-detect email address (got 'ubuntu@nitrcce.(none)') install(ok): /home/ubuntu/containers (dataset) ubuntu@nitrcce:~$ git config --global user.email "nobody@example.com" ubuntu@nitrcce:~$ git config --global user.name "Your Name" ubuntu@nitrcce:~$ cd containers/ ubuntu@nitrcce:~/containers$ bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```') ... > datalad containers-run -n containers/bids-mriqc --input sourcedata --output . '{inputs}' '{outputs}' participant group [INFO ] Making sure inputs are available (this may take some time) get(ok): sourcedata/sub-02/anat/sub-02_T1w.nii.gz (file) [from s3-PUBLIC...] get(ok): sourcedata/sub-02/anat/sub-02_inplaneT2.nii.gz (file) [from s3-PUBLIC...] get(ok): sourcedata/sub-02/func/sub-02_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...] get(ok): sourcedata/sub-13/anat/sub-13_T1w.nii.gz (file) [from s3-PUBLIC...] get(ok): sourcedata/sub-13/anat/sub-13_inplaneT2.nii.gz (file) [from s3-PUBLIC...] get(ok): sourcedata/sub-13/func/sub-13_task-rhymejudgment_bold.nii.gz (file) [from s3-PUBLIC...] get(ok): containers/images/bids/bids-mriqc--0.15.1.sing (file) [from origin...] [INFO ] == Command start (output follows) ===== 2020-12-15 15:35:23,242 mriqc:IMPORTANT Running MRIQC version 0.15.1: * BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata. * Output folder: /tmp/repro-3vNHTmV/ds000003-qc. * Analysis levels: participant, group. Process Process-2: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 400, in init_mriqc ignore=['derivatives', 'sourcedata', r'^\..*']) File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 185, in __init__ self._validate_root() File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 318, in _validate_root raise ValueError("BIDS root does not exist: %s" % self.root) ValueError: BIDS root does not exist: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata [INFO ] == Command exit (modification check follows) ===== [INFO ] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -d . -r -F .git/COMMIT_EDITMSG' CommandError: 'containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata' '' participant group' failed with exitcode 1 under /tmp/repro-3vNHTmV/ds000003-qcmore specific reproducing + not-reproducing with direct invocation when already in the container
```shell ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ singularity exec -W /tmp/singtmp.PacrEs -B /tmp/singtmp.PacrEs/tmp:/tmp -B /tmp/singtmp.PacrEs/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing bash bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8) bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ export | grep SING declare -x SINGULARITY_APPNAME="" declare -x SINGULARITY_CONTAINER="/tmp/repro-3vNHTmV/ds000003-qc/containers/images/bids/bids-mriqc--0.15.1.sing" declare -x SINGULARITY_NAME="bids-mriqc--0.15.1.sing" bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ ls /tmp/repro-3vNHTmV/ds000003-qc/sourcedata CHANGES README dataset_description.json participants.tsv sub-02 sub-13 task-rhymejudgment_bold.json bidsapp@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ /.singularity.d/runscript sourcedata '' participant group 2020-12-15 15:40:41,456 mriqc:IMPORTANT Running MRIQC version 0.15.1: * BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata. * Output folder: /tmp/repro-3vNHTmV/ds000003-qc. * Analysis levels: participant, group. 201215-15:40:47,656 nipype.workflow INFO: Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz'] 2020-12-15 15:40:47,656 nipype.workflow:INFO Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz'] ^CException ignored in:so it feels like bind mount of
/tmp/repro-3vNHTmV/ds000003-qcdid not happen... but reason seems to be somewhere due to our singularity_cmd halper since it fails with it:it fails with it
```shell ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ bash -x containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata' . participant group + set -eu ++ readlink -f containers/scripts/singularity_cmd + thisfile=/tmp/repro-3vNHTmV/ds000003-qc/containers/scripts/singularity_cmd ++ dirname /tmp/repro-3vNHTmV/ds000003-qc/containers/scripts/singularity_cmd + thisdir=/tmp/repro-3vNHTmV/ds000003-qc/containers/scripts ++ dirname /tmp/repro-3vNHTmV/ds000003-qc/containers/scripts + updir=/tmp/repro-3vNHTmV/ds000003-qc/containers + BHOME=/tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME + cmd=run + shift + '[' -n '' ']' ++ mktemp -d -t singtmp.XXXXXX + tmpdir=/tmp/singtmp.ywiVAe + info 'created temp dir /tmp/singtmp.ywiVAe' + : + trap 'rm -fr "$tmpdir" && info "removed temp dir $tmpdir"' exit + pass_git_config user.name 'ReproNim User' + var=user.name + default='ReproNim User' ++ git config user.name + value='Your Name' + for attempt in {1..5} + git config -f /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME/.gitconfig user.name + break + pass_git_config user.email nobody@example.com + var=user.email + default=nobody@example.com ++ git config user.email + value=nobody@example.com + for attempt in {1..5} + git config -f /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME/.gitconfig user.email + break + SARGS=(-e -B "$PWD" -H "$BHOME" --pwd "$PWD" "$@") + need_no_c= + for d in "$PWD" "$updir" + '[' repro-3vNHTmV/ds000003-qc '!=' /tmp/repro-3vNHTmV/ds000003-qc ']' + info 'Creating /tmp/repro-3vNHTmV/ds000003-qc under /tmp/singtmp.ywiVAe' + : + mkdir -p /tmp/singtmp.ywiVAe//tmp/repro-3vNHTmV/ds000003-qc /tmp/singtmp.ywiVAe/var/tmp + need_no_c=1 + for d in "$PWD" "$updir" + '[' repro-3vNHTmV/ds000003-qc/containers '!=' /tmp/repro-3vNHTmV/ds000003-qc/containers ']' + info 'Creating /tmp/repro-3vNHTmV/ds000003-qc/containers under /tmp/singtmp.ywiVAe' + : + mkdir -p /tmp/singtmp.ywiVAe//tmp/repro-3vNHTmV/ds000003-qc/containers /tmp/singtmp.ywiVAe/var/tmp + need_no_c=1 + '[' -z 1 ']' + SARGS=(-B "$tmpdir/tmp:/tmp" -B "$tmpdir/var/tmp:/var/tmp" "${SARGS[@]}") + hash singularity + '[' -z '' ']' + singularity run -W /tmp/singtmp.ywiVAe -B /tmp/singtmp.ywiVAe/tmp:/tmp -B /tmp/singtmp.ywiVAe/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing sourcedata . participant group 2020-12-15 15:46:55,399 mriqc:IMPORTANT Running MRIQC version 0.15.1: * BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata. * Output folder: /tmp/repro-3vNHTmV/ds000003-qc. * Analysis levels: group, participant. Process Process-2: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap self.run() File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run self._target(*self._args, **self._kwargs) File "/usr/local/miniconda/lib/python3.7/site-packages/mriqc/bin/mriqc_run.py", line 400, in init_mriqc ignore=['derivatives', 'sourcedata', r'^\..*']) File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 185, in __init__ self._validate_root() File "/usr/local/miniconda/lib/python3.7/site-packages/bids/layout/layout.py", line 318, in _validate_root raise ValueError("BIDS root does not exist: %s" % self.root) ValueError: BIDS root does not exist: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata + rm -fr /tmp/singtmp.ywiVAe + info 'removed temp dir /tmp/singtmp.ywiVAe' + : ```but works if I just go with the direct singularity command it runs:
```shell ubuntu@nitrcce:/tmp/repro-3vNHTmV/ds000003-qc$ singularity run -W /tmp/singtmp.ywiVAe -B /tmp/singtmp.ywiVAe/tmp:/tmp -B /tmp/singtmp.ywiVAe/var/tmp:/var/tmp -e -B /tmp/repro-3vNHTmV/ds000003-qc -H /tmp/repro-3vNHTmV/ds000003-qc/containers/binds/HOME --pwd /tmp/repro-3vNHTmV/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing sourcedata '' participant group 2020-12-15 15:47:50,992 mriqc:IMPORTANT Running MRIQC version 0.15.1: * BIDS dataset path: /tmp/repro-3vNHTmV/ds000003-qc/sourcedata. * Output folder: /tmp/repro-3vNHTmV/ds000003-qc. * Analysis levels: participant, group. 201215-15:47:57,125 nipype.workflow INFO: Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz'] 2020-12-15 15:47:57,125 nipype.workflow:INFO Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz'] ^C ```so the reason is really is having that
/tmp/singtmp.ywiVAe(and subdirs) which bind mount first and then elderly singularity refuses to bind mount /tmp/repro-3vNHTmV/ds000003-qc on top of it...quick&dirty workaround is to place TMPDIR somewhere else but
/tmp, e.g.mkdir -p ~/tmpand provide it while running the example, e.g.TMPDIR=~/tmp bash <(sed -n -e '/^ *#!/,/^```$/p' README.md | grep -v '```')then we do not bother to create/bind mount those additional paths
```shell ubuntu@nitrcce:~/tmp/repro-wgnyF4J/ds000003-qc$ TMPDIR=~/tmp bash -x containers/scripts/singularity_cmd run containers/images/bids/bids-mriqc--0.15.1.sing 'sourcedata' . participant group + set -eu ++ readlink -f containers/scripts/singularity_cmd + thisfile=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts/singularity_cmd ++ dirname /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts/singularity_cmd + thisdir=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts ++ dirname /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/scripts + updir=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers + BHOME=/home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME + cmd=run + shift + '[' -n '' ']' ++ mktemp -d -t singtmp.XXXXXX + tmpdir=/home/ubuntu/tmp/singtmp.yq0pJg + info 'created temp dir /home/ubuntu/tmp/singtmp.yq0pJg' + : + trap 'rm -fr "$tmpdir" && info "removed temp dir $tmpdir"' exit + pass_git_config user.name 'ReproNim User' + var=user.name + default='ReproNim User' ++ git config user.name + value='Your Name' + for attempt in {1..5} + git config -f /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME/.gitconfig user.name + break + pass_git_config user.email nobody@example.com + var=user.email + default=nobody@example.com ++ git config user.email + value=nobody@example.com + for attempt in {1..5} + git config -f /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME/.gitconfig user.email + break + SARGS=(-e -B "$PWD" -H "$BHOME" --pwd "$PWD" "$@") + need_no_c= + for d in "$PWD" "$updir" + '[' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc '!=' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc ']' + for d in "$PWD" "$updir" + '[' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers '!=' /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers ']' + '[' -z '' ']' + SARGS=(-c "${SARGS[@]}") + hash singularity + '[' -z '' ']' + singularity run -W /home/ubuntu/tmp/singtmp.yq0pJg -c -e -B /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc -H /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/containers/binds/HOME --pwd /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc containers/images/bids/bids-mriqc--0.15.1.sing sourcedata . participant group 2020-12-15 16:24:57,803 mriqc:IMPORTANT Running MRIQC version 0.15.1: * BIDS dataset path: /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc/sourcedata. * Output folder: /home/ubuntu/tmp/repro-wgnyF4J/ds000003-qc. * Analysis levels: participant, group. 201215-16:25:04,31 nipype.workflow INFO: Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz'] 2020-12-15 16:25:04,031 nipype.workflow:INFO Building anatomical MRIQC workflow, datasets list: ['sub-02/anat/sub-02_T1w.nii.gz', 'sub-13/anat/sub-13_T1w.nii.gz'] ... ```proper fix is yet to be worked out...