Closed hcp4715 closed 4 years ago
yarikoptic
commented
4 years ago what exactly do you mean by "lock all the files" (we do have some locking but that should not anyhow interfere with external processes)? Or do you mean it makes those files "read-only"? copy/pasted output of an error would have been useful
yarikoptic
commented
4 years ago we do seems to make those files readonly indeed, but I would take it as a feature I guess, which seems had been there for a while:
15940b72 (Matteo Visconti di Oleggio Castello 2018-02-08 14:18:44 -0500 496) if outname and op.exists(outname):
1467d072 (Mathias Goncalves 2017-10-20 18:22:58 -0400 497) set_readonly(outname)modifying raw files inplace makes them non raw really, and I guess that is what we tried to avoid - any side effects. If someone (a tool) wants to modify those files inplace -- it better takes care about permissions. It could also just remove them before writing a "replacement" in place. I believe in some defacing scripts we even moved that original file under sourcedata/ (and not shared publicly) to retain the original source file (while also retaining its RO attribute).
Overall, I consider it a feature and a good practice to adhere to by default. Someone might want to add an option to disable it explicitly. If external tool is to modify that file -- just chmod it explicitly first or tune that tool to do that.
hcp4715
commented
4 years ago Hi, @yarikoptic , Thanks for your quick response.
Or do you mean it makes those files "read-only"?
Yes, that's what I meant.
I have a special situation other than deface: my T1 data have a FOV of 300 300, which need to be cropped to 256 256 before using fmriprep.
I agree that making files RO is a good practice, also, thanks to your great suggestion about moving the original file under sourcedata/
Below is the error when I run BIDSonym immediate after using heudiconv 0.8.0:
hcp4715@Duo:~/Data/RepDopa/Nipy$ singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base \
> /home/hcp4715/docker_images/bidsonym-lastest.simg \
> /base/BIDS_test/ \
> participant \
> --participant_label 002 \
> --deid pydeface \
> --del_meta 'InstitutionAddress'
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
This dataset appears to be BIDS compatible.
Summary: Available Tasks: Available Modalities:
87 Files, 2.78GB exp T1w
2 - Subjects fam T2w
2 - Sessions rest dwi
TODO: full task name for fam bold
TODO: full task name for exp events
TODO: full task name for rest fieldmap
If you have any questions, please post on https://neurostars.org/tags/bids.
200527-22:04:45,166 nipype.utils INFO:
Running nipype version 1.5.0-rc1 (latest: 1.4.2)
200527-22:04:45,518 nipype.workflow INFO:
Workflow deface_wf settings: ['check', 'execution', 'logging', 'monitoring']
200527-22:04:45,521 nipype.workflow INFO:
Running serially.
200527-22:04:45,521 nipype.workflow INFO:
[Node] Setting-up "deface_wf.pydeface" in "/tmp/tmpa5e9cryr/deface_wf/pydeface".
200527-22:04:45,523 nipype.workflow INFO:
[Node] Running "pydeface" ("nipype.interfaces.utility.wrappers.Function")
--------------
pydeface 2.0.0
--------------
Previous output will be overwritten.
Temporary files:
/tmp/tmpvd4ebuvs.mat
/tmp/tmp7961y0nz.nii.gz
Defacing...
/base/BIDS_test/sub-002/ses-d1/anat/sub-002_ses-d1_T1w.nii.gz
200527-22:04:46,583 nipype.utils INFO:
Running nipype version 1.5.0-rc1 (latest: 1.4.2)
Traceback (most recent call last):
File "/opt/miniconda-latest/envs/bidsonym/bin/pydeface", line 11, in <module>
load_entry_point('pydeface==2.0.0', 'console_scripts', 'pydeface')()
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/__main__.py", line 95, in main
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/utils.py", line 126, in deface_image
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/filebasedimages.py", line 334, in to_filename
self.to_file_map()
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/analyze.py", line 1035, in to_file_map
hdrf = hdr_fh.get_prepare_fileobj(mode='wb')
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/fileholders.py", line 70, in get_prepare_fileobj
obj = ImageOpener(self.filename, *args, **kwargs)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 111, in __init__
self.fobj = opener(fileish, *args, **kwargs)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 51, in _gzip_open
gzip_file = gzip.GzipFile(filename, mode, compresslevel)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/gzip.py", line 163, in __init__
fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-002/ses-d1/anat/sub-002_ses-d1_T1w.nii.gz'
200527-22:05:56,935 nipype.workflow WARNING:
Storing result file without outputs
200527-22:05:56,935 nipype.workflow WARNING:
[Node] Error on "deface_wf.pydeface" (/tmp/tmpa5e9cryr/deface_wf/pydeface)
200527-22:05:56,937 nipype.workflow ERROR:
Node pydeface failed to run on host NIC-Duo.
200527-22:05:56,937 nipype.workflow ERROR:
Saving crash info to /home/hcp4715/Data/RepDopa/Nipy/crash-20200527-220556-hcp4715-pydeface-c1b12251-b23e-4e31-b53a-704a5b839103.pklz
Traceback (most recent call last):
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
node.run(updatehash=updatehash)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
result = self._run_interface(execute=True)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
return self._run_command(execute)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
result = self._interface.run(cwd=outdir)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 397, in run
runtime = self._run_interface(runtime)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
out = function_handle(**args)
File "<string>", line 9, in pydeface_cmd
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/subprocess.py", line 311, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pydeface', '/base/BIDS_test/sub-002/ses-d1/anat/sub-002_ses-d1_T1w.nii.gz', '--out', '/base/BIDS_test/sub-002/ses-d1/anat/sub-002_ses-d1_T1w.nii.gz', '--force']' returned non-zero exit status 1.
200527-22:05:56,938 nipype.workflow INFO:
***********************************
200527-22:05:56,938 nipype.workflow ERROR:
could not run node: deface_wf.pydeface
200527-22:05:56,938 nipype.workflow INFO:
crashfile: /home/hcp4715/Data/RepDopa/Nipy/crash-20200527-220556-hcp4715-pydeface-c1b12251-b23e-4e31-b53a-704a5b839103.pklz
200527-22:05:56,938 nipype.workflow INFO:
***********************************
Traceback (most recent call last):
File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 11, in <module>
load_entry_point('bidsonym', 'console_scripts', 'bidsonym')()
File "/home/bm/bidsonym/run_deeid.py", line 111, in run_deeid
run_pydeface(T1_file, T1_file)
File "/home/bm/bidsonym/defacing_algorithms.py", line 32, in run_pydeface
deface_wf.run()
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
runner.run(execgraph, updatehash=updatehash, config=self.config)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 70, in run
report_nodes_not_run(notrun)
File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/tools.py", line 98, in report_nodes_not_run
("Workflow did not execute cleanly. " "Check log for details")
RuntimeError: Workflow did not execute cleanly. Check log for details
Summary
The new version of heudiconv will lock all the files, which caused a problem: it can’t be modified by BIDSonym. which means that I need to change the permission of the T1 file and then run BIDSonym or crop the T1 data.
I tried to disable this function but can't find a solution in the tutorial, which is still based on 0.5.4 Just had a brief chat with @PeerHerholz, we thought it's better to open a new issue on this.
Here is the code I used:
Platform details:
Choose one:
[x] Local environment Ubuntu 20.04
[x] Container singularity
Heudiconv version: 0.8.0
This question has been posted on neurostar: https://neurostars.org/t/file-lock-of-heudicov-0-8-0-and-bidsonym/6801