Pydeface writing permission error for python egg cache

[m-petersen]

Describe the bug

Currently I am trying to implement bidsonym in a preprocessing pipeline using v0.0.4 in a singularity container on our local HPC. A submitted job doesn't have writing permission for the /home partition. However, when using bidsonym with pydeface it apparently wants to write in the python egg cache directory localised on /home yielding the error message below. I tried to export PYTHON_EGG_CACHE to a writable directory (/scratch) without success.

Do you have any ideas how to address that?

Happy to provide further information.

Cheers, Marvin

To Reproduce CMD=" singularity run --cleanenv --userns --no-home\ -B $CLONE:/bids \ -B $CLONE_DCM_DIR:/dcm \ -B $CLONE_TMPDIR:/tmp \ $ENV_DIR/bidsonym-v0.0.4\ /bids participant \ --participant_label ${1} \ --deid pydeface \ --brainextraction bet --bet_frac 0.5 \ --del_meta '' \ "

Expected behavior To perform pydeface without an error.

OS/environment (please provide precise information on your OS/environment): CentOS 6 HPC bidsonym singularity container v0.0.4 from dockerhub CMD=" singularity run --cleanenv --userns --no-home\ -B $CLONE:/bids \ -B $CLONE_DCM_DIR:/dcm \ -B $CLONE_TMPDIR:/tmp \ $ENV_DIR/bidsonym-v0.0.4\ /bids participant \ --participant_label ${1} \ --deid pydeface \ --brainextraction bet --bet_frac 0.5 \ --del_meta '' \ "

Additional context 0: -------------- 0: pydeface 2.0.0 0: -------------- local:1/1/100%/207.0s 0: Traceback (most recent call last): 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 1209, in get_cache_path 0: _bypass_ensure_directory(target_path) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 3198, in _bypass_ensure_directory 0: _bypass_ensure_directory(dirname) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 3198, in _bypass_ensure_directory 0: _bypass_ensure_directory(dirname) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 3198, in _bypass_ensure_directory 0: _bypass_ensure_directory(dirname) 0: [Previous line repeated 2 more times] 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 3200, in _bypass_ensure_directory 0: mkdir(dirname, 0o755) 0: OSError: [Errno 30] Read-only file system: '/home/fatx405' 0: 0: During handling of the above exception, another exception occurred: 0: 0: Traceback (most recent call last): 0: File "/opt/miniconda-latest/envs/bidsonym/bin/pydeface", line 33, in 0: sys.exit(load_entry_point('pydeface==2.0.0', 'console_scripts', 'pydeface')()) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/main.py", line 95, in main 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/utils.py", line 86, in deface_image 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/utils.py", line 17, in initial_checks 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 1144, in resource_filename 0: self, resource_name 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 1754, in get_resource_filename 0: return self._extract_resource(manager, zip_path) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 1783, in _extract_resource 0: self.egg_name, self._parts(zip_path) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 1211, in get_cache_path 0: self.extraction_error() 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pkg_resources/init.py", line 1191, in extraction_error 0: raise err 0: pkg_resources.ExtractionError: Can't extract file(s) to egg cache 0: 0: The following error occurred while trying to extract file(s) 0: to the Python egg cache: 0: 0: [Errno 30] Read-only file system: '/home/fatx405' 0: 0: The Python egg cache directory is currently set to: 0: 0: /home/fatx405/.cache/Python-Eggs 0: 0: Perhaps your account does not have write access to this directory? 0: You can change the cache directory by setting the PYTHON_EGG_CACHE 0: environment variable to point to an accessible directory. 0: 0: 210624-13:36:32,738 nipype.workflow WARNING: 0: Storing result file without outputs 0: 210624-13:36:32,741 nipype.workflow WARNING: 0: [Node] Error on "deface_wf.pydeface" (/tmp/tmpecm1gxor/deface_wf/pydeface) 0: 210624-13:36:32,746 nipype.workflow ERROR: 0: Node pydeface failed to run on host node317. 0: 210624-13:36:32,746 nipype.workflow ERROR: 0: Saving crash info to /scratch/fatx405.1426534/10812127/clone/crash-20210624-133632-fatx405-pydeface-45b27c12-5de7-498e-8f87-71608a4c2a8e.pklz 0: Traceback (most recent call last): 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run 0: node.run(updatehash=updatehash) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run 0: result = self._run_interface(execute=True) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface 0: return self._run_command(execute) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command 0: result = self._interface.run(cwd=outdir) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 434, in run 0: runtime = self._run_interface(runtime) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface 0: out = function_handle(args) 0: File "", line 9, in pydeface_cmd 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/subprocess.py", line 311, in check_call 0: raise CalledProcessError(retcode, cmd) 0: subprocess.CalledProcessError: Command '['pydeface', '/bids/sourcedata/bidsonym/sub-10812127/sub-10812127_ses-1_T1w.nii.gz', '--out', '/bids/sub-10812127/ses-1/anat/sub-10812127_ses-1_T1w.nii.gz', '--force']' returned non-zero exit status 1. 0: 0: 210624-13:36:32,747 nipype.workflow INFO: 0: ** 0: 210624-13:36:32,747 nipype.workflow ERROR: 0: could not run node: deface_wf.pydeface 0: 210624-13:36:32,747 nipype.workflow INFO: 0: crashfile: /scratch/fatx405.1426534/10812127/clone/crash-20210624-133632-fatx405-pydeface-45b27c12-5de7-498e-8f87-71608a4c2a8e.pklz 0: 210624-13:36:32,747 nipype.workflow INFO: 0: 0: /opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/bids/layout/models.py:152: FutureWarning: The 'extension' entity currently excludes the leading dot ('.'). As of version 0.14.0, it will include the leading dot. To suppress this warning and include the leading dot, use bids.config.set_option('extension_initial_dot', True). 0: FutureWarning) 0: Traceback (most recent call last): 0: File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in 0: sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')()) 0: File "/home/bm/bidsonym/run_deeid.py", line 143, in run_deeid 0: run_pydeface(source_t1w, T1_file) 0: File "/home/bm/bidsonym/defacing_algorithms.py", line 32, in run_pydeface 0: deface_wf.run() 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run 0: runner.run(execgraph, updatehash=updatehash, config=self.config) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 70, in run 0: report_nodes_not_run(notrun) 0: File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/tools.py", line 98, in report_nodes_not_run 0: ("Workflow did not execute cleanly. " "Check log for details") 0: RuntimeError: Workflow did not execute cleanly. Check log for details 0: [INFO] == Command exit (modification check follows) ===== 0: [INFO] The command had a non-zero exit code. If this is expected, you can save the changes with 'datalad save -d . -r -F .git/COMMIT_EDITMSG' 0: CommandError: 'singularity run --cleanenv --userns --no-home -B /scratch/fatx405.1426534/10812127/clone:/bids -B /scratch/fatx405.1426534/10812127/clone/sourcedata:/dcm -B /scratch/fatx405.1426534/10812127/tmp/:/tmp /work/fatx405/projects/2105_test_proc/envs/bidsonym-v0.0.4 /bids participant --participant_label 10812127 --deid pydeface --brainextraction bet --bet_frac 0.5 --del_meta ''"'"''"'"''' failed with exitcode 1 under /scratch/fatx405.1426534/10812127/clone

[PeerHerholz]

Ahoi hoi @m-petersen,

thx a lot for reporting this! Hm, interesting...I've never encountered this problem, there's always something new hehe. Two questions just to be sure:

1. would you mind sharing why you're using the --userns & --no-home flags?
2. when exporting PYTHON_EGG_CACHE, does the subsequent bidsonym run show the updated directory or still /home/fatx405/.cache/Python-Eggs?

Cheers, Peer

[m-petersen]

Dear Peer,

thanks for your reply.

1. --userns is recommended by our cluster admins for our local singularity install to get certain privileges inside the container. --no-home was just a quick solution attempt although I wasn't expecting it to be successful. Without --no-home the same error occurs.
2. Actually, the output of my previous post is from a run where PYTHON_EGG_CACHE was set to /scratch where I have writing permission. There the egg cache is still /home/fatx405/.cache/Python-Eggs.

The full command (wrapped in datalad run) is:

CMD="
             singularity run --cleanenv --userns --no-home\
             -B $CLONE:/bids \
             -B $CLONE_DCM_DIR:/dcm \
             -B $CLONE_TMPDIR:/tmp \
             $ENV_DIR/bidsonym-v0.0.4\
             /bids participant \
             --participant_label ${1} \
             --deid pydeface \
             --brainextraction bet --bet_frac 0.5 \
             --del_meta '' \
         "
         # Set python egg cache directory for bidsonym
         export PYTHON_EGG_CACHE=$SCRATCH_DIR

         datalad run -m '${PIPE_ID}-bidsonym' \
            --explicit \
            --output "$CLONE/sub-$1" \
            --input "$CLONE/sub-$1/" \
            $CMD

[PeerHerholz]

Ahoi hoi @m-petersen,

thx for the additional information. Which singularity version are you using? One thing you could try is setting environment variables within the container using the SINGULARITYENV_ magic, e.g.:

export SINGULARITYENV_PYTHON_EGG_CACHE=$SCRATCH_DIR

Great to see that you're using DataLad! You could also look into its container extension called datalad containers. BIDSonym should soon be added there and thus also within ReproNim/containers.

Cheers, Peer

[m-petersen]

export SINGULARITYENV_PYTHON_EGG_CACHE=$SCRATCH indeed changed the environment variable in the container. But the permission error remained because of the singularity container still lacking writing permissions in $SCRATCH. Changing the environment variable to /tmp where I have mounted $SCRATCH in the container worked. Thanks a lot for your help!

And thanks for the recommendation. I actually already work with the datalad containers extension but sometimes have some syntax problems and then switch to datalad run.

Cheers, Marvin

[PeerHerholz]

Hi @m-petersen,

coolio, glad it works now and thanks again for bringing this up! I would like to note that the export SINGULARITYENV_ idea came from @emdupre, so all credit should go to her!

Cheers, Peer