read-only input files should not prevent the creation of output replacement

[yarikoptic]

Please see initial issue https://github.com/nipy/heudiconv/issues/457#issuecomment-634909468 : in heudiconv, even without using --datalad option, we make produced "raw" files read-only to prevent accidental unintended changes. That report points to the fact that bidsonym tries to produce output ile "in place", which then fails. I would recommend to simply remove original file before replacing it. Or may be even add an option to move original file under sourcedata/ where it actually nicely belongs ;) that would also work nicely with datalad datasets (unless source data is a submodule, so moving git annex'ed file/symlink wouldn't work)

[PeerHerholz]

Hi @yarikoptic and @hcp4715,

thx @hcp4715 for discovering and posting this issue, as we briefly talked about in our call, this is something very important we need to address.

@yarikoptic, we and some other folks talked about this a while back but didn't really follow up on it. My bad, sorry for that. If I remember correctly, the problem is the status of source data, raw data and derived data and their differentiation wrt data sharing. In order to allow for data sharing, we need to do our best in terms of deidentifying the data. As this includes alterations of the image and .json files within the top level directory, this would mean that we don't share raw data but derived data. Comparable to what you suggested BIDSonym currently copies the original raw data to sourcedata/bidsonym to keep them in case something goes wrong during the deidentification (e.g. too stringent defacing) and thus prevent the necessity of re-converting from DICOM again. Thus, I could alter the behavior so that original raw data is moved and not copied to sourcedata/bidsonym and the deidentified data is saved to the top level directory. As also discussed before, we could integrate DataLad accordingly. As long as this is explicitly stated, it might work (even though it's suboptimal of course).

What do folks think about this?

Also tagging others that might have valuable input here (sorry for that): @satra, @effigies, @franklin-feingold and @sappelhoff.

[effigies]

I don't really see a distinction between:

1) Copy original image into sourcedata/ and overwrite with defaced version 2) Move original image into sourcedata/ and write defaced version with original filename

Except that the latter avoids some permission issues.

So I'm feeling like I'm missing some subtlety here...

[PeerHerholz]

Thx for the reply @effigies. You're absolutely right in that the output would be identical as the original image would be in sourcedata/ and the deidentified one in bids root.

I think it's more of an conceptual discussion regarding terms as mentioned above, but this/should be targeted somewhere else anyway... Regarding the specific function (copy vs. move): we talked about that during the OHBM brainhack last year, but not in depth or focused in any way. As both lead to the same result and to ensure easy compatibility with heudiconv I would vote for option two.

[yarikoptic]

original raw data to sourcedata/bidsonym

FWIW, although it is "moved" there, it is not really produced by bidsonym, so not sure why to use bidsonym/ prefix there. Why not to just place it under the original dataset path straight under sourcedata/. E.g. in reproin heuristic we place all dicoms in exactly matching patch under sourcedata/, only with a different extension. Make is really easy to find the "sourcedata/" for a corresponding file without first needing to discover where possibly it was stashed. E.g. in https://github.com/dbic/QA for sub-emmet/ses-20180508/anat/sub-emmet_ses-20180508_acq-MPRAGE_T1w.nii.gz its source DICOM data is placed under sourcedata/sub-emmet/ses-20180508/anat/sub-emmet_ses-20180508_acq-MPRAGE_T1w.dicom.tgz so it is really easy to find it.

Sure thing some utilities could start puking at first by discovering bids-like filenames under sourcedata/ - but it would only serve us all well, since the tools would need to be fixed.

[PeerHerholz]

We went with /bidsonym as the file naming/structure was one of the addressed issues for which no solution was provided back at the time. Thus, it acted/acts as a temporary solution. One problem is that the file extension would be the same and the non-deidentified status needs to be expressed respectively which is currently done by adding no_deid to the filename (e.g. sub-01_Tw1.nii.gz --> sub-01_Tw1_no_deid.nii.gz). Following your suggestion this would lead to a sourcedata/ version that includes both, the DICOM before conversion to BIDS and the no-deid version of anat images and .json sidecar files, right?

[yarikoptic]

Sorry, I am not following where .json comes from under sourcedata/ . Since there is no standard under sourcedata - I guess any suffix like _no_deid is ok and would make it explicit and not bids compliant so less likely to confuse bids tools sneaking around sourcedata by a bug ;-)

[PeerHerholz]

Sorry for not being more precise here, BIDSonym also checks/evaluates the .json files wrt information that should potentially not be there/makes it easier to identify participants. Thus, those would be moved and potentially altered as well.

[yarikoptic]

Thanks! Anything particular in output from heudiconv ever triggers need in changes in those json files?

[PeerHerholz]

Sorry for the late reply. I think this depends on a given data sharing protocol, as well as what's set within the sequence. For example, InstitutionAddress or other fields along these lines. In order to evaluate if potentially sensitive information is present, BIDSonym checks both, json files and image headers, writing the respective information to a tsv, adding a column indicated if information might be "problematic" (e.g. if name is included). The option for altering files however only supports changing the json but not the image files of course. It's obviously a lower priority (hopefully) than the defacing, but I wanted to support this functionality to ensure the highest possible degree of deindentification, especially wrt our endeavours within the GDPR.

[yarikoptic]

just ran into this "old" issue again, and wondered if it could be resolved one way or another ;-)

[PeerHerholz]

fixed via this commit.

[hcp4715]

Hi, @PeerHerholz , I tried this today, the permission issue still persists.

My code:

singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base \
>     /home/hcp4715/docker_images/bidsonym-0.0.2.simg \
>     /base/BIDS/ \
>     participant \
>     --participant_label 001 \
>     --deid pydeface \
>     --brainextraction bet --bet_frac 0.5 \
>     --del_meta 'InstitutionAddress'

Output:

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

This dataset appears to be BIDS compatible.

        Summary:                  Available Tasks:                     Available Modalities: 
        911 Files, 35.69GB        exp                                  T1w                   
        14 - Subjects             fam                                  T2w                   
        3 - Sessions              rest                                 dwi                   
                                  TODO: full task name for exp         bold                  
                                  TODO: full task name for fam         events                
                                  TODO: full task name for rest        fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/bids/layout/models.py:102: FutureWarning: The 'extension' entity currently excludes the leading dot ('.'). As of version 0.14.0, it will include the leading dot. To suppress this warning and include the leading dot, use `bids.config.set_option('extension_initial_dot', True)`.
  FutureWarning)
200819-22:30:42,517 nipype.workflow INFO:
     Workflow brainextraction_wf settings: ['check', 'execution', 'logging', 'monitoring']
200819-22:30:42,520 nipype.workflow INFO:
     Running serially.
200819-22:30:42,520 nipype.workflow INFO:
     [Node] Setting-up "brainextraction_wf.bet" in "/tmp/tmp7cyoc072/brainextraction_wf/bet".
200819-22:30:42,522 nipype.workflow INFO:
     [Node] Running "bet" ("nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /base/BIDS/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz /base/BIDS/sourcedata/bidsonym/sub-001/sub-001_space-native_brainmask.nii.gz -f 0.50 -m
200819-22:30:47,310 nipype.workflow INFO:
     [Node] Finished "brainextraction_wf.bet".
200819-22:30:47,334 nipype.workflow INFO:
     Workflow deface_wf settings: ['check', 'execution', 'logging', 'monitoring']
200819-22:30:47,336 nipype.workflow INFO:
     Running serially.
200819-22:30:47,336 nipype.workflow INFO:
     [Node] Setting-up "deface_wf.pydeface" in "/tmp/tmpn3zbgnz5/deface_wf/pydeface".
200819-22:30:47,337 nipype.workflow INFO:
     [Node] Running "pydeface" ("nipype.interfaces.utility.wrappers.Function")
--------------
pydeface 2.0.0
--------------
Previous output will be overwritten.
Temporary files:
  /tmp/tmp_jn8io5g.mat
  /tmp/tmpfd__fhs_.nii.gz
Defacing...
  /base/BIDS/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz
Defaced image saved as:
  /base/BIDS/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz
Cleaning up...
Finished.
200819-22:31:53,139 nipype.workflow INFO:
     [Node] Finished "deface_wf.pydeface".
working on 001
found the following meta-data files:
/base/BIDS/task-rest_bold.json
/base/BIDS/dataset_description.json
/base/BIDS/participants.json
/base/BIDS/task-exp_bold.json
/base/BIDS/task-fam_bold.json
/base/BIDS/sub-001/ses-d1/sub-001_ses-d1_scans.json
/base/BIDS/sub-001/ses-d1/fmap/sub-001_ses-d1_magnitude2.json
/base/BIDS/sub-001/ses-d1/fmap/sub-001_ses-d1_magnitude1.json
/base/BIDS/sub-001/ses-d1/fmap/sub-001_ses-d1_phasediff.json
/base/BIDS/sub-001/ses-d1/anat/sub-001_ses-d1_T2w.json
/base/BIDS/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.json
/base/BIDS/sub-001/ses-d1/dwi/sub-001_ses-d1_dwi.json
/base/BIDS/sub-001/ses-d1/func/sub-001_ses-d1_task-fam_bold.json
/base/BIDS/sub-001/ses-d1/func/sub-001_ses-d1_task-rest_run-2_bold.json
/base/BIDS/sub-001/ses-d1/func/sub-001_ses-d1_task-rest_run-1_bold.json
/base/BIDS/sub-001/ses-d1/func/sub-001_ses-d1_task-exp_bold.json
/base/BIDS/sub-001/ses-d3/sub-001_ses-d3_scans.json
/base/BIDS/sub-001/ses-d3/func/sub-001_ses-d3_task-exp_bold.json
/base/BIDS/sub-001/ses-d3/func/sub-001_ses-d3_task-rest_run-1_bold.json
/base/BIDS/sub-001/ses-d2/sub-001_ses-d2_scans.json
/base/BIDS/sub-001/ses-d2/fmap/sub-001_ses-d2_phasediff.json
/base/BIDS/sub-001/ses-d2/fmap/sub-001_ses-d2_magnitude2.json
/base/BIDS/sub-001/ses-d2/fmap/sub-001_ses-d2_magnitude1.json
/base/BIDS/sub-001/ses-d2/func/sub-001_ses-d2_task-exp_bold.json
/base/BIDS/sub-001/ses-d2/func/sub-001_ses-d2_task-rest_run-3_bold.json
/base/BIDS/sub-001/ses-d2/func/sub-001_ses-d2_task-rest_run-4_bold.json
/base/BIDS/sub-001/ses-d2/func/sub-001_ses-d2_task-rest_run-1_bold.json
/base/BIDS/sub-001/ses-d2/func/sub-001_ses-d2_task-rest_run-2_bold.json
the following fields will be deleted:
InstitutionAddress
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 130, in run_deeid
    del_meta_data(args.bids_dir, subject_label, list_field_del)
  File "/home/bm/bidsonym/utils.py", line 133, in del_meta_data
    with open(meta_file, 'w') as json_output_file:
PermissionError: [Errno 13] Permission denied: '/base/BIDS/sub-001/ses-d1/fmap/sub-001_ses-d1_phasediff.json'

A related the issue again, after running the BIDSonym, it added one line to the ./participants.json and make this file not BIDS.

hcp4715@NIC-Duo:~$ singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base     /home/hcp4715/docker_images/bidsonym-0.0.2.simg     /base/BIDS/     participant     --participant_label 001     --deid pydeface     --brainextraction bet --bet_frac 0.5     --del_meta 'InstitutionAddress' 

output:

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

    1: [ERR] Invalid JSON file. The file is not formatted according the schema. (code: 55 - JSON_SCHEMA_VALIDATION_ERROR)
        ./participants.json
            Evidence: ['InstitutionAddress'] should be object

[PeerHerholz]

Hi @hcp4715,

thanks for testing again and reporting this issue. Hm, the tests worked on my side (classic comment, hehe). Are the json files copied to sourcedata/? Re the second problem: I just used the BIDS validator on my test dataset after running BIDSonym and it was valid. Thus, I assume your manual edit create a problem. Could you elaborate on that?

[hcp4715]

Hi, @PeerHerholz thanks for your quick response!

I re-tested the deface function, as follow:

Step 1, convert dicom to bids using heudiconv

sublist=("001")

for subject in "${sublist[@]}"; 
do
    for session in {1..3}; 
    do
    sesId="d"
    sesId+="${session}" #$(echo "${session: -2}")  # get the last two characters of session folder
        # take care of the space in the [ ] after if
    heuristicFile="/base/heuristic/"                   # directory of the heuristic files
    if [ ${sesId} == "d1" ]; then
        heuristic='heuristic_d1.py'
    elif [ ${sesId} == "d2" ]; then
        heuristic='heuristic_d2.py'
    elif [ ${sesId} == "d3" ]; then
        heuristic='heuristic_d3.py'
    fi

    heuristicFile+="${heuristic}"  # get the direct of heuristic file
    #echo ${heuristicFile}

    singularity run --cleanenv \
    -B /home/hcp4715/Data/RepDopa/Nipy:/base \
    /home/hcp4715/docker_images/heudiconv-0.8.0.simg \
    -d /base/Dicom/sub-{subject}/ses-{session}/*/* \
    -o /base/BIDS_test/ \
    -f ${heuristicFile} \
    -s ${subject} \
    -ss ${sesId} \
    -c dcm2niix \
    -b --overwrite --minmeta
    done
done

Step 2, validate the BIDS data.

so, the data are in bids format.

Step 3, try BIDSonym

singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base \
    /home/hcp4715/docker_images/bidsonym-0.0.2.simg \
    /base/BIDS_test/ \
    participant \
    --participant_label 001 \
    --deid pydeface \
    --brainextraction bet --bet_frac 0.5 \
    --del_meta 'InstitutionAddress'

Here, I had the old permission error

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

This dataset appears to be BIDS compatible.

        Summary:                Available Tasks:                     Available Modalities: 
        67 Files, 2.45GB        exp                                  T1w                   
        1 - Subject             fam                                  T2w                   
        3 - Sessions            rest                                 dwi                   
                                TODO: full task name for exp         bold                  
                                TODO: full task name for fam         events                
                                TODO: full task name for rest        fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/bids/layout/models.py:102: FutureWarning: The 'extension' entity currently excludes the leading dot ('.'). As of version 0.14.0, it will include the leading dot. To suppress this warning and include the leading dot, use `bids.config.set_option('extension_initial_dot', True)`.
  FutureWarning)
200820-23:33:03,652 nipype.workflow INFO:
     Workflow brainextraction_wf settings: ['check', 'execution', 'logging', 'monitoring']
200820-23:33:03,654 nipype.workflow INFO:
     Running serially.
200820-23:33:03,654 nipype.workflow INFO:
     [Node] Setting-up "brainextraction_wf.bet" in "/tmp/tmppfhi9b5a/brainextraction_wf/bet".
200820-23:33:03,655 nipype.workflow INFO:
     [Node] Running "bet" ("nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz /base/BIDS_test/sourcedata/bidsonym/sub-001/sub-001_space-native_brainmask.nii.gz -f 0.50 -m
200820-23:33:08,430 nipype.workflow INFO:
     [Node] Finished "brainextraction_wf.bet".
200820-23:33:08,447 nipype.workflow INFO:
     Workflow deface_wf settings: ['check', 'execution', 'logging', 'monitoring']
200820-23:33:08,449 nipype.workflow INFO:
     Running serially.
200820-23:33:08,449 nipype.workflow INFO:
     [Node] Setting-up "deface_wf.pydeface" in "/tmp/tmp41qi2qaf/deface_wf/pydeface".
200820-23:33:08,450 nipype.workflow INFO:
     [Node] Running "pydeface" ("nipype.interfaces.utility.wrappers.Function")
--------------
pydeface 2.0.0
--------------
Previous output will be overwritten.
Temporary files:
  /tmp/tmpw3cipvs4.mat
  /tmp/tmpc9ovwynj.nii.gz
Defacing...
  /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/pydeface", line 33, in <module>
    sys.exit(load_entry_point('pydeface==2.0.0', 'console_scripts', 'pydeface')())
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/__main__.py", line 95, in main
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/utils.py", line 126, in deface_image
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/filebasedimages.py", line 334, in to_filename
    self.to_file_map()
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/analyze.py", line 1033, in to_file_map
    hdrf = hdr_fh.get_prepare_fileobj(mode='wb')
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/fileholders.py", line 70, in get_prepare_fileobj
    obj = ImageOpener(self.filename, *args, **kwargs)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 114, in __init__
    self.fobj = opener(fileish, *args, **kwargs)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 54, in _gzip_open
    gzip_file = gzip.GzipFile(filename, mode, compresslevel)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz'
200820-23:34:12,233 nipype.workflow WARNING:
     Storing result file without outputs
200820-23:34:12,234 nipype.workflow WARNING:
     [Node] Error on "deface_wf.pydeface" (/tmp/tmp41qi2qaf/deface_wf/pydeface)
200820-23:34:12,235 nipype.workflow ERROR:
     Node pydeface failed to run on host NIC-Duo.
200820-23:34:12,235 nipype.workflow ERROR:
     Saving crash info to /home/hcp4715/crash-20200820-233412-hcp4715-pydeface-f60eb337-ed9e-41ad-9daa-8dc81cadb894.pklz
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
    node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
    out = function_handle(**args)
  File "<string>", line 9, in pydeface_cmd
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pydeface', '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz', '--out', '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz', '--force']' returned non-zero exit status 1.

200820-23:34:12,235 nipype.workflow INFO:
     ***********************************
200820-23:34:12,235 nipype.workflow ERROR:
     could not run node: deface_wf.pydeface
200820-23:34:12,236 nipype.workflow INFO:
     crashfile: /home/hcp4715/crash-20200820-233412-hcp4715-pydeface-f60eb337-ed9e-41ad-9daa-8dc81cadb894.pklz
200820-23:34:12,236 nipype.workflow INFO:
     ***********************************
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 127, in run_deeid
    run_pydeface(T1_file, T1_file)
  File "/home/bm/bidsonym/defacing_algorithms.py", line 32, in run_pydeface
    deface_wf.run()
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 70, in run
    report_nodes_not_run(notrun)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/tools.py", line 98, in report_nodes_not_run
    ("Workflow did not execute cleanly. " "Check log for details")
RuntimeError: Workflow did not execute cleanly. Check log for details

Step 4, manually change the permission of the T1 data and re-run the above code:

(base) hcp4715@NIC-Duo:~$ singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base     /home/hcp4715/docker_images/bidsonym-0.0.2.simg     /base/BIDS_test/     participant     --participant_label 001     --deid pydeface     --brainextraction bet --bet_frac 0.5     --del_meta 'InstitutionAddress'
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

This dataset appears to be BIDS compatible.

        Summary:                Available Tasks:                     Available Modalities: 
        67 Files, 2.45GB        exp                                  T1w                   
        1 - Subject             fam                                  T2w                   
        3 - Sessions            rest                                 dwi                   
                                TODO: full task name for fam         bold                  
                                TODO: full task name for exp         events                
                                TODO: full task name for rest        fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/bids/layout/models.py:102: FutureWarning: The 'extension' entity currently excludes the leading dot ('.'). As of version 0.14.0, it will include the leading dot. To suppress this warning and include the leading dot, use `bids.config.set_option('extension_initial_dot', True)`.
  FutureWarning)
200820-23:41:35,698 nipype.workflow INFO:
     Workflow brainextraction_wf settings: ['check', 'execution', 'logging', 'monitoring']
200820-23:41:35,707 nipype.workflow INFO:
     Running serially.
200820-23:41:35,708 nipype.workflow INFO:
     [Node] Setting-up "brainextraction_wf.bet" in "/tmp/tmp1m068d3i/brainextraction_wf/bet".
200820-23:41:35,710 nipype.workflow INFO:
     [Node] Running "bet" ("nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz /base/BIDS_test/sourcedata/bidsonym/sub-001/sub-001_space-native_brainmask.nii.gz -f 0.50 -m
200820-23:41:40,591 nipype.workflow INFO:
     [Node] Finished "brainextraction_wf.bet".
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 126, in run_deeid
    copy_no_deid(subject_label, args.bids_dir, T1_file)
  File "/home/bm/bidsonym/utils.py", line 30, in copy_no_deid
    copy(T1_file, os.path.join(path, outfile))
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/shutil.py", line 245, in copy
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/shutil.py", line 121, in copyfile
    with open(dst, 'wb') as fdst:
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sourcedata/bidsonym/sub-001/sub-001_ses-d1_T1w_no_deid.nii.gz'

Step 5, delete files in /base/BIDS_test/sourcedata/bidsonym/sub-001/, and re-run the above code:

...
...
the following fields will be deleted:
InstitutionAddress
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 130, in run_deeid
    del_meta_data(args.bids_dir, subject_label, list_field_del)
  File "/home/bm/bidsonym/utils.py", line 133, in del_meta_data
    with open(meta_file, 'w') as json_output_file:
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-001/ses-d1/fmap/sub-001_ses-d1_phasediff.json'

Step 6, manually change the permission of files in /base/BIDS_test/sub-001/ses-d1/fmap/ and delete the temporary files, and tried the above code again:

(base) hcp4715@NIC-Duo:~$ singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base     /home/hcp4715/docker_images/bidsonym-0.0.2.simg     /base/BIDS_test/     participant     --participant_label 001     --deid pydeface     --brainextraction bet --bet_frac 0.5     --del_meta 'InstitutionAddress'
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

    1: [ERR] Invalid JSON file. The file is not formatted according the schema. (code: 55 - JSON_SCHEMA_VALIDATION_ERROR)
        ./participants.json
            Evidence: ['InstitutionAddress'] should be object

    Please visit https://neurostars.org/search?q=JSON_SCHEMA_VALIDATION_ERROR for existing conversations about this issue.

        Summary:                Available Tasks:                     Available Modalities: 
        67 Files, 2.45GB        exp                                  T1w                   
        1 - Subject             fam                                  T2w                   
        3 - Sessions            rest                                 dwi                   
                                TODO: full task name for exp         bold                  
                                TODO: full task name for fam         events                
                                TODO: full task name for rest        fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 80, in run_deeid
    validate_input_dir(exec_env, args.bids_dir, args.participant_label)
  File "/home/bm/bidsonym/utils.py", line 274, in validate_input_dir
    subprocess.check_call(['bids-validator', bids_dir, '-c', temp.name])
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['bids-validator', PosixPath('/base/BIDS_test'), '-c', '/tmp/tmpg10_d_5y']' returned non-zero exit status 1.

Step 8, check BIDS:

The participants.json file's content is as below:

{
    "participant_id": {
        "Description": "Participant identifier"
    },
    "age": {
        "Description": "Age in years (TODO - verify) as in the initial session, might not be correct for other sessions"
    },
    "sex": {
        "Description": "self-rated by participant, M for male/F for female (TODO: verify)"
    },
    "group": {
        "Description": "(TODO: adjust - by default everyone is in control group)"
    },
    "InstitutionAddress": "deleted_by_bidsonym"
}

Step 9, delete the "InstitutionAddress": "deleted_by_bidsonym" part, and re-try bids validator, passed!

Step 10, try bidsonym again:

Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 130, in run_deeid
    del_meta_data(args.bids_dir, subject_label, list_field_del)
  File "/home/bm/bidsonym/utils.py", line 133, in del_meta_data
    with open(meta_file, 'w') as json_output_file:
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T2w.json'

Step 11, manually change the permission of all files in /base/BIDS_test/sub-001/ses-d1/anat/ and also repeated Step 5 and Step 9. rerun BIDSonym.

the following fields will be deleted:
InstitutionAddress
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 130, in run_deeid
    del_meta_data(args.bids_dir, subject_label, list_field_del)
  File "/home/bm/bidsonym/utils.py", line 133, in del_meta_data
    with open(meta_file, 'w') as json_output_file:
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-001/ses-d1/dwi/sub-001_ses-d1_dwi.json'

Step 12, manually change the permission of all files in /base/BIDS_test/sub-001/ses-d1/dwi/ and also repeated Step 5 and Step 9. rerun BIDSonym.

the following fields will be deleted:
InstitutionAddress
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 130, in run_deeid
    del_meta_data(args.bids_dir, subject_label, list_field_del)
  File "/home/bm/bidsonym/utils.py", line 133, in del_meta_data
    with open(meta_file, 'w') as json_output_file:
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-001/ses-d1/func/sub-001_ses-d1_task-fam_bold.json'

Step 13, give up today, I guess I need to manually change the permission of all json files.

In short, seems that the updated version of BIDSonym requires to access all anatomical data and all json files of the subjects, instead of T1 only in the older version? Am I using the same version as you did? Here is how I pull the image:

singularity build --sandbox /home/hcp4715/docker_images/bidsonym-0.0.2.simg  docker:peerherholz/bidsonym:v0.0.2

[PeerHerholz]

Hm, a lot of stuff going on there. Before we dig deeper into all of this, could you try pulling the image again? I just triggered a new build as I think something went wrong there. I'm sorry that you're having a hard time running BIDSonym.

[hcp4715]

hi, @PeerHerholz I re-puledl the image and repeated the Step 12 in my last post, the same error occured.

[PeerHerholz]

Hi @hcp4715,

could you try the running BIDSonym on a freshly converted dataset again, posting the output here? Sorry for the hustle!

[hcp4715]

Hi, @PeerHerholz , I just ran BIDSonym on a newly converted dataset, without changing the permission, below is the output:

base) hcp4715@thu-duo:~$ singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base \
>     /home/hcp4715/containers/bidsonym-0.0.2.simg \
>     /base/BIDS_test/ \
>     participant \
>     --participant_label 001 \
>     --deid pydeface \
>     --brainextraction bet --bet_frac 0.5 \
>     --del_meta 'InstitutionAddress'
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

This dataset appears to be BIDS compatible.

        Summary:                  Available Tasks:                     Available Modalities: 
        36 Files, 858.36MB        exp                                  T1w                   
        1 - Subject               fam                                  T2w                   
        1 - Session               rest                                 dwi                   
                                  TODO: full task name for exp         bold                  
                                  TODO: full task name for fam         events                
                                  TODO: full task name for rest        fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/bids/layout/models.py:102: FutureWarning: The 'extension' entity currently excludes the leading dot ('.'). As of version 0.14.0, it will include the leading dot. To suppress this warning and include the leading dot, use `bids.config.set_option('extension_initial_dot', True)`.
  FutureWarning)
200825-00:05:02,495 nipype.workflow INFO:
     Workflow brainextraction_wf settings: ['check', 'execution', 'logging', 'monitoring']
200825-00:05:02,500 nipype.workflow INFO:
     Running serially.
200825-00:05:02,500 nipype.workflow INFO:
     [Node] Setting-up "brainextraction_wf.bet" in "/tmp/tmp5ihcli0m/brainextraction_wf/bet".
200825-00:05:02,502 nipype.workflow INFO:
     [Node] Running "bet" ("nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz /base/BIDS_test/sourcedata/bidsonym/sub-001/sub-001_space-native_brainmask.nii.gz -f 0.50 -m
200825-00:05:08,831 nipype.workflow INFO:
     [Node] Finished "brainextraction_wf.bet".
200825-00:05:08,856 nipype.workflow INFO:
     Workflow deface_wf settings: ['check', 'execution', 'logging', 'monitoring']
200825-00:05:08,858 nipype.workflow INFO:
     Running serially.
200825-00:05:08,858 nipype.workflow INFO:
     [Node] Setting-up "deface_wf.pydeface" in "/tmp/tmptggliutv/deface_wf/pydeface".
200825-00:05:08,860 nipype.workflow INFO:
     [Node] Running "pydeface" ("nipype.interfaces.utility.wrappers.Function")
--------------
pydeface 2.0.0
--------------
Previous output will be overwritten.
Temporary files:
  /tmp/tmpv_j12ymi.mat
  /tmp/tmpnjhw4y2d.nii.gz
Defacing...
  /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/pydeface", line 33, in <module>
    sys.exit(load_entry_point('pydeface==2.0.0', 'console_scripts', 'pydeface')())
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/__main__.py", line 95, in main
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/utils.py", line 126, in deface_image
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/filebasedimages.py", line 334, in to_filename
    self.to_file_map()
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/analyze.py", line 1033, in to_file_map
    hdrf = hdr_fh.get_prepare_fileobj(mode='wb')
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/fileholders.py", line 70, in get_prepare_fileobj
    obj = ImageOpener(self.filename, *args, **kwargs)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 114, in __init__
    self.fobj = opener(fileish, *args, **kwargs)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 54, in _gzip_open
    gzip_file = gzip.GzipFile(filename, mode, compresslevel)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz'
200825-00:06:35,327 nipype.workflow WARNING:
     Storing result file without outputs
200825-00:06:35,328 nipype.workflow WARNING:
     [Node] Error on "deface_wf.pydeface" (/tmp/tmptggliutv/deface_wf/pydeface)
200825-00:06:35,330 nipype.workflow ERROR:
     Node pydeface failed to run on host thu-duo.
200825-00:06:35,330 nipype.workflow ERROR:
     Saving crash info to /home/hcp4715/crash-20200825-000635-hcp4715-pydeface-957ee162-9b1f-4da6-9792-cdf9921bf9cf.pklz
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
    node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
    out = function_handle(**args)
  File "<string>", line 9, in pydeface_cmd
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pydeface', '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz', '--out', '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz', '--force']' returned non-zero exit status 1.

200825-00:06:35,331 nipype.workflow INFO:
     ***********************************
200825-00:06:35,331 nipype.workflow ERROR:
     could not run node: deface_wf.pydeface
200825-00:06:35,331 nipype.workflow INFO:
     crashfile: /home/hcp4715/crash-20200825-000635-hcp4715-pydeface-957ee162-9b1f-4da6-9792-cdf9921bf9cf.pklz
200825-00:06:35,331 nipype.workflow INFO:
     ***********************************
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 127, in run_deeid
    run_pydeface(T1_file, T1_file)
  File "/home/bm/bidsonym/defacing_algorithms.py", line 32, in run_pydeface
    deface_wf.run()
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 70, in run
    report_nodes_not_run(notrun)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/tools.py", line 98, in report_nodes_not_run
    ("Workflow did not execute cleanly. " "Check log for details")
RuntimeError: Workflow did not execute cleanly. Check log for details

[PeerHerholz]

Thx @hcp4715,

I'm still not quite sure what's going on, as your output indicates that the files in the BIDS root will be de-identified. However, as I changed the code to move the original files to sourcedata and de-identify from there, the output should state Defacing ... /base/BIDS_test/sourcedata/bidsonym/... . I edited some build related problems and tested it locally with a newly build version of master. I hate to be that tool developer again, but "it works on my end". Could you maybe try again using the mater version of the image (docker pull peerherholz/bidsonym:latest) and paste the output again? Sorry!

[hcp4715]

Hi @PeerHerholz, no need to apologize. I think your adjustment worked, we are almost there.

I re-converted a participant's data, and used BIDSonym to de-identify the data, no error message:

(base) hcp4715@thu-duo:~$ singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base \
>     /home/hcp4715/containers/bidsonym-latest.simg \
>     /base/BIDS_test/ \
>     participant \
>     --participant_label 001 \
>     --deid pydeface \
>     --brainextraction bet --bet_frac 0.5 \
>     --del_meta 'InstitutionAddress'
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

This dataset appears to be BIDS compatible.

        Summary:                  Available Tasks:                     Available Modalities: 
        36 Files, 858.36MB        exp                                  T1w                   
        1 - Subject               fam                                  T2w                   
        1 - Session               rest                                 dwi                   
                                  TODO: full task name for exp         bold                  
                                  TODO: full task name for rest        events                
                                  TODO: full task name for fam         fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/bids/layout/models.py:102: FutureWarning: The 'extension' entity currently excludes the leading dot ('.'). As of version 0.14.0, it will include the leading dot. To suppress this warning and include the leading dot, use `bids.config.set_option('extension_initial_dot', True)`.
  FutureWarning)
Processing data from 1 sessions:
['d1']
200825-23:32:49,273 nipype.workflow INFO:
     Workflow brainextraction_wf settings: ['check', 'execution', 'logging', 'monitoring']
200825-23:32:49,279 nipype.workflow INFO:
     Running serially.
200825-23:32:49,279 nipype.workflow INFO:
     [Node] Setting-up "brainextraction_wf.bet" in "/tmp/tmpythed90m/brainextraction_wf/bet".
200825-23:32:49,281 nipype.workflow INFO:
     [Node] Running "bet" ("nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz /base/BIDS_test/sourcedata/bidsonym/sub-001/sub-001_ses-d1_space-native_brainmask.nii.gz -f 0.50 -m
200825-23:32:55,143 nipype.workflow INFO:
     [Node] Finished "brainextraction_wf.bet".
200825-23:32:55,197 nipype.workflow INFO:
     Workflow deface_wf settings: ['check', 'execution', 'logging', 'monitoring']
200825-23:32:55,199 nipype.workflow INFO:
     Running serially.
200825-23:32:55,199 nipype.workflow INFO:
     [Node] Setting-up "deface_wf.pydeface" in "/tmp/tmpc5l380jy/deface_wf/pydeface".
200825-23:32:55,201 nipype.workflow INFO:
     [Node] Running "pydeface" ("nipype.interfaces.utility.wrappers.Function")
--------------
pydeface 2.0.0
--------------
Temporary files:
  /tmp/tmpxq4pqszf.mat
  /tmp/tmpmao44hfq.nii.gz
Defacing...
  /base/BIDS_test/sourcedata/bidsonym/sub-001/sub-001_ses-d1_T1w.nii.gz
Defaced image saved as:
  /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz
Cleaning up...
Finished.
200825-23:34:15,11 nipype.workflow INFO:
     [Node] Finished "deface_wf.pydeface".
working on 001
found the following meta-data files:
/base/BIDS_test/task-exp_bold.json
/base/BIDS_test/participants.json
/base/BIDS_test/task-fam_bold.json
/base/BIDS_test/task-rest_bold.json
/base/BIDS_test/dataset_description.json
/base/BIDS_test/sub-001/ses-d1/sub-001_ses-d1_scans.json
/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T2w.json
/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.json
/base/BIDS_test/sub-001/ses-d1/dwi/sub-001_ses-d1_dwi.json
/base/BIDS_test/sub-001/ses-d1/fmap/sub-001_ses-d1_magnitude2.json
/base/BIDS_test/sub-001/ses-d1/fmap/sub-001_ses-d1_phasediff.json
/base/BIDS_test/sub-001/ses-d1/fmap/sub-001_ses-d1_magnitude1.json
/base/BIDS_test/sub-001/ses-d1/func/sub-001_ses-d1_task-fam_bold.json
/base/BIDS_test/sub-001/ses-d1/func/sub-001_ses-d1_task-rest_run-2_bold.json
/base/BIDS_test/sub-001/ses-d1/func/sub-001_ses-d1_task-rest_run-1_bold.json
/base/BIDS_test/sub-001/ses-d1/func/sub-001_ses-d1_task-exp_bold.json
the following fields will be deleted:
InstitutionAddress

But, again, the BIDS validation failed.

Below is the validation results after BIDSonym:

{
    "participant_id": {
        "Description": "Participant identifier"
    },
    "age": {
        "Description": "Age in years (TODO - verify) as in the initial session, might not be correct for other sessions"
    },
    "sex": {
        "Description": "self-rated by participant, M for male/F for female (TODO: verify)"
    },
    "group": {
        "Description": "(TODO: adjust - by default everyone is in control group)"
    },
    "InstitutionAddress": "deleted_by_bidsonym"
}

Below is the results before BIDSonym:

{
  "participant_id": {
    "Description": "Participant identifier"
  },
  "age": {
    "Description": "Age in years (TODO - verify) as in the initial session, might not be correct for other sessions"
  },
  "sex": {
    "Description": "self-rated by participant, M for male/F for female (TODO: verify)"
  },
  "group": {
    "Description": "(TODO: adjust - by default everyone is in control group)"
  }
}

[PeerHerholz]

Hi @hcp4715,

I adapted the del_meta_data function to first evaluate if the field/key that should be deleted is actually there. However, regarding the other errors (TR and TE) I have no idea where they are coming from, as BIDSonym doesn't touch or change this information. Could you try again (from a freshly converted dataset) with either the latest or v0.0.2 version?

[hcp4715]

Hi, @PeerHerholz

I just tried, old permission issue again.

hcp4715@thu-duo:~$ singularity run --cleanenv -B /home/hcp4715/Data/RepDopa/Nipy:/base \
>     /home/hcp4715/containers/bidsonym-0.0.2.simg \
>     /base/BIDS_test/ \
>     participant \
>     --participant_label 001 \
>     --deid pydeface \
>     --brainextraction bet --bet_frac 0.5 \
>     --del_meta 'InstitutionAddress'
Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
bids-validator@1.5.4

This dataset appears to be BIDS compatible.

        Summary:                  Available Tasks:                     Available Modalities: 
        36 Files, 858.36MB        exp                                  T1w                   
        1 - Subject               fam                                  T2w                   
        1 - Session               rest                                 dwi                   
                                  TODO: full task name for exp         bold                  
                                  TODO: full task name for fam         events                
                                  TODO: full task name for rest        fieldmap              

    If you have any questions, please post on https://neurostars.org/tags/bids.

/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/bids/layout/models.py:102: FutureWarning: The 'extension' entity currently excludes the leading dot ('.'). As of version 0.14.0, it will include the leading dot. To suppress this warning and include the leading dot, use `bids.config.set_option('extension_initial_dot', True)`.
  FutureWarning)
200903-22:40:22,77 nipype.workflow INFO:
     Workflow brainextraction_wf settings: ['check', 'execution', 'logging', 'monitoring']
200903-22:40:22,82 nipype.workflow INFO:
     Running serially.
200903-22:40:22,82 nipype.workflow INFO:
     [Node] Setting-up "brainextraction_wf.bet" in "/tmp/tmp57rvu07j/brainextraction_wf/bet".
200903-22:40:22,84 nipype.workflow INFO:
     [Node] Running "bet" ("nipype.interfaces.fsl.preprocess.BET"), a CommandLine Interface with command:
bet /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz /base/BIDS_test/sourcedata/bidsonym/sub-001/sub-001_space-native_brainmask.nii.gz -f 0.50 -m
200903-22:40:28,243 nipype.workflow INFO:
     [Node] Finished "brainextraction_wf.bet".
200903-22:40:28,267 nipype.workflow INFO:
     Workflow deface_wf settings: ['check', 'execution', 'logging', 'monitoring']
200903-22:40:28,269 nipype.workflow INFO:
     Running serially.
200903-22:40:28,270 nipype.workflow INFO:
     [Node] Setting-up "deface_wf.pydeface" in "/tmp/tmpjvhdwkh5/deface_wf/pydeface".
200903-22:40:28,271 nipype.workflow INFO:
     [Node] Running "pydeface" ("nipype.interfaces.utility.wrappers.Function")
--------------
pydeface 2.0.0
--------------
Previous output will be overwritten.
Temporary files:
  /tmp/tmplbejm0i1.mat
  /tmp/tmpddo7bw63.nii.gz
Defacing...
  /base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/pydeface", line 33, in <module>
    sys.exit(load_entry_point('pydeface==2.0.0', 'console_scripts', 'pydeface')())
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/__main__.py", line 95, in main
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/pydeface-2.0.0-py3.6.egg/pydeface/utils.py", line 126, in deface_image
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/filebasedimages.py", line 334, in to_filename
    self.to_file_map()
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/analyze.py", line 1033, in to_file_map
    hdrf = hdr_fh.get_prepare_fileobj(mode='wb')
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/fileholders.py", line 70, in get_prepare_fileobj
    obj = ImageOpener(self.filename, *args, **kwargs)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 114, in __init__
    self.fobj = opener(fileish, *args, **kwargs)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nibabel/openers.py", line 54, in _gzip_open
    gzip_file = gzip.GzipFile(filename, mode, compresslevel)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/gzip.py", line 163, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
PermissionError: [Errno 13] Permission denied: '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz'
200903-22:41:49,625 nipype.workflow WARNING:
     Storing result file without outputs
200903-22:41:49,625 nipype.workflow WARNING:
     [Node] Error on "deface_wf.pydeface" (/tmp/tmpjvhdwkh5/deface_wf/pydeface)
200903-22:41:49,627 nipype.workflow ERROR:
     Node pydeface failed to run on host thu-duo.
200903-22:41:49,628 nipype.workflow ERROR:
     Saving crash info to /home/hcp4715/crash-20200903-224149-hcp4715-pydeface-d0bae1fe-03d8-4c37-b5d0-fc4a6e296bfc.pklz
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
    node.run(updatehash=updatehash)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
    out = function_handle(**args)
  File "<string>", line 9, in pydeface_cmd
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['pydeface', '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz', '--out', '/base/BIDS_test/sub-001/ses-d1/anat/sub-001_ses-d1_T1w.nii.gz', '--force']' returned non-zero exit status 1.

200903-22:41:49,629 nipype.workflow INFO:
     ***********************************
200903-22:41:49,629 nipype.workflow ERROR:
     could not run node: deface_wf.pydeface
200903-22:41:49,629 nipype.workflow INFO:
     crashfile: /home/hcp4715/crash-20200903-224149-hcp4715-pydeface-d0bae1fe-03d8-4c37-b5d0-fc4a6e296bfc.pklz
200903-22:41:49,629 nipype.workflow INFO:
     ***********************************
Traceback (most recent call last):
  File "/opt/miniconda-latest/envs/bidsonym/bin/bidsonym", line 33, in <module>
    sys.exit(load_entry_point('bidsonym', 'console_scripts', 'bidsonym')())
  File "/home/bm/bidsonym/run_deeid.py", line 127, in run_deeid
    run_pydeface(T1_file, T1_file)
  File "/home/bm/bidsonym/defacing_algorithms.py", line 32, in run_pydeface
    deface_wf.run()
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/linear.py", line 70, in run
    report_nodes_not_run(notrun)
  File "/opt/miniconda-latest/envs/bidsonym/lib/python3.6/site-packages/nipype/pipeline/plugins/tools.py", line 98, in report_nodes_not_run
    ("Workflow did not execute cleanly. " "Check log for details")
RuntimeError: Workflow did not execute cleanly. Check log for details

also, seems that BIDSonym created two folders in my home folder: deface_wf and brainextraction_wf, both have four files: d3.js, graph.json, graph1.json, index.html. I doubt these two folders created in a wrong directoy.

[PeerHerholz]

Hm, I'm pretty sure that's not the latest version as pydeface is trying to deface the images in the BIDS root folder and this was changed a few commits back being the main point of this issue. Did you pull the images again from Docker/Singularity hub?

Re the folders: this is nipype output and is there based on your path mapping.

[hcp4715]

Thanks for your quick feedback, @PeerHerholz I did pull the image again. To make everything clear, I cleaned the singularity cache and try again. latest version can run without any error, the content in participants.json is OK now. but version 0.0.2 doesn't work, permission error persists.

But even after the BIDSonym worked without error, BIDS validation still threw two errors and one more warning.

Error 1: RepetitionTime changed

Error 2: phasediff json file changed.

Warning: Slice time not defined

I checked the json files.

Error 1 Before: { "AcquisitionMatrixPE": 72, "AcquisitionNumber": 1, "AcquisitionTime": "17:02:50.090000", "BandwidthPerPixelPhaseEncode": 24.802, "BaseResolution": 84, "CoilString": "C:HEA;HEP", "ConversionSoftware": "dcm2niix", "ConversionSoftwareVersion": "v1.0.20190410 GCC6.3.0", "DerivedVendorReportedEchoSpacing": 0.000559991, "DeviceSerialNumber": "35333", "DwellTime": 2.3e-06, "EchoTime": 0.029, "EffectiveEchoSpacing": 0.000559991, "FlipAngle": 56, "ImageComments": "Unaliased_MB4_PE3", "ImageOrientationPatientDICOM": [1, 2e-16, 0, -2e-16, 0.990024, -0.140901], "ImageType": [ "ORIGINAL", "PRIMARY", "M", "MB", "ND", "MOSAIC" ], "ImagingFrequency": 123.209, "InPlanePhaseEncodingDirectionDICOM": "COL", "InstitutionAddress": "XXX_District_DE_XXX", "InstitutionalDepartmentName": "Department", "MRAcquisitionType": "2D", "MagneticFieldStrength": 3, "Manufacturer": "Siemens", "ManufacturersModelName": "TrioTim", "Modality": "MR", "MultibandAccelerationFactor": 4, "PartialFourier": 1, "PatientPosition": "HFS", "PercentPhaseFOV": 85.7143, "PhaseEncodingDirection": "j-", "PhaseEncodingSteps": 72, "PhaseResolution": 1, "PixelBandwidth": 2588, "ProcedureStepDescription": "MR-Studie_Dopamine", "ProtocolName": "fmri_Tag1_mb4_TR_1000", "PulseSequenceDetails": "%CustomerSeq%_cmrr_mbep2d_bold", "ReceiveCoilName": "32Ch_Head", "ReconMatrixPE": 72, "RepetitionTime": 1, "SAR": 0.100207, "ScanOptions": "FS", "ScanningSequence": "EP", "SequenceName": "epfid2d1_72", "SequenceVariant": "SK_SS", "SeriesDescription": "fmri_Tag1_mb4_TR_1000", "SeriesNumber": 4, "ShimSetting": [7155, 10407, 22426, 553, -71, 223, 264, -49], "SliceThickness": 2.5, "SliceTiming": [0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675, 0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675, 0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675, 0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675], "SoftwareVersions": "syngo_MR_B17", "SpacingBetweenSlices": 2.5, "TaskName": "fam", "TotalReadoutTime": 0.0397593, "TxRefAmp": 338.281}

After: { "AcquisitionMatrixPE": 72, "AcquisitionNumber": 1, "AcquisitionTime": "17:02:50.090000", "BandwidthPerPixelPhaseEncode": 24.802, "BaseResolution": 84, "CoilString": "C:HEA;HEP", "ConversionSoftware": "dcm2niix", "ConversionSoftwareVersion": "v1.0.20190410 GCC6.3.0", "DerivedVendorReportedEchoSpacing": 0.000559991, "DeviceSerialNumber": "35333", "DwellTime": 2.3e-06, "EchoTime": 0.029, "EffectiveEchoSpacing": 0.000559991, "FlipAngle": 56, "ImageComments": "Unaliased_MB4_PE3", "ImageOrientationPatientDICOM": [ 1, 2e-16, 0, -2e-16, 0.990024, -0.140901 ], "ImageType": [ "ORIGINAL", "PRIMARY", "M", "MB", "ND", "MOSAIC" ], "ImagingFrequency": 123.209, "InPlanePhaseEncodingDirectionDICOM": "COL", "InstitutionAddress": "deleted_by_bidsonym", "InstitutionalDepartmentName": "Department", "MRAcquisitionType": "2D", "MagneticFieldStrength": 3, "Manufacturer": "Siemens", "ManufacturersModelName": "TrioTim", "Modality": "MR", "MultibandAccelerationFactor": 4, "PartialFourier": 1, "PatientPosition": "HFS", "PercentPhaseFOV": 85.7143, "PhaseEncodingDirection": "j-", "PhaseEncodingSteps": 72, "PhaseResolution": 1, "PixelBandwidth": 2588, "ProcedureStepDescription": "MR-Studie_Dopamine", "ProtocolName": "fmri_Tag1_mb4_TR_1000", "PulseSequenceDetails": "%CustomerSeq%_cmrr_mbep2d_bold", "ReceiveCoilName": "32Ch_Head", "ReconMatrixPE": 72, "RepetitionTime": 1, "SAR": 0.100207, "ScanOptions": "FS", "ScanningSequence": "EP", "SequenceName": "epfid2d1_72", "SequenceVariant": "SK_SS", "SeriesDescription": "fmri_Tag1_mb4_TR_1000", "SeriesNumber": 4, "ShimSetting": [ 7155, 10407, 22426, 553, -71, 223, 264, -49 ], "SliceThickness": 2.5, "SliceTiming": [ 0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675, 0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675, 0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675, 0, 0.535, 0.0675, 0.6025, 0.1325, 0.6675, 0.2, 0.735, 0.2675, 0.8025, 0.335, 0.87, 0.4, 0.9375, 0.4675 ], "SoftwareVersions": "syngo_MR_B17", "SpacingBetweenSlices": 2.5, "TaskName": "exp", ## this change is unexpected and might cause problem in late analysis. "TotalReadoutTime": 0.0397593, "TxRefAmp": 338.281 }

Error2.

Before:

{
  "AcquisitionMatrixPE": 64,
  "AcquisitionNumber": 1,
  "AcquisitionTime": "17:32:18.367500",
  "BaseResolution": 64,
  "CoilString": "C:HEA;HEP",
  "ConversionSoftware": "dcm2niix",
  "ConversionSoftwareVersion": "v1.0.20190410  GCC6.3.0",
  "DeviceSerialNumber": "35333",
  "DwellTime": 3.62e-05,
  "EchoNumber": 2,
  "EchoTime": 0.00735,
  "EchoTime1": 0.00489,
  "EchoTime2": 0.00735,
  "FlipAngle": 60,
  "ImageOrientationPatientDICOM": [1, 2e-16, 0, -2e-16, 0.990024, -0.140901],
  "ImageType": [
    "ORIGINAL",
    "PRIMARY",
    "P",
    "ND"
],
  "ImagingFrequency": 123.209,
  "InPlanePhaseEncodingDirectionDICOM": "COL",
  "InstitutionAddress": "XXXX_street_XXX",
  "InstitutionalDepartmentName": "Department",
  "MRAcquisitionType": "2D",
  "MagneticFieldStrength": 3,
  "Manufacturer": "Siemens",
  "ManufacturersModelName": "TrioTim",
  "Modality": "MR",
  "PartialFourier": 1,
  "PatientPosition": "HFS",
  "PercentPhaseFOV": 100,
  "PhaseEncodingDirection": "j-",
  "PhaseEncodingSteps": 64,
  "PhaseResolution": 1,
  "PixelBandwidth": 216,
  "ProcedureStepDescription": "MR-Studie_XXX",
  "ProtocolName": "gre_field_mapping",
  "PulseSequenceDetails": "%SiemensSeq%_gre_field_mapping",
  "ReceiveCoilName": "32Ch_Head",
  "ReconMatrixPE": 64,
  "RepetitionTime": 1.03,
  "SAR": 0.13381,
  "ScanningSequence": "GR",
  "SequenceName": "_fm2d2r",
  "SequenceVariant": "SP",
  "SeriesDescription": "gre_field_mapping",
  "SeriesNumber": 9,
  "ShimSetting": [7155, 10408, 22429, 574, -74, 239, 263, -55],
  "SliceThickness": 3,
  "SoftwareVersions": "syngo_MR_B17",
  "SpacingBetweenSlices": 3,
  "TxRefAmp": 338.281}

After:

{
    "filename": {
        "Description": "Name of the nifti file"
    },
    "acq_time": {
        "LongName": "Acquisition time",
        "Description": "Acquisition time of the particular scan"
    },
    "operator": {
        "Description": "Name of the operator"
    },
    "randstr": {
        "LongName": "Random string",
        "Description": "md5 hash of UIDs"
    }
}

Didn't check the warning about slice timing, but BIDSonym indeed changed the json files.

[PeerHerholz]

Tbh: I have no idea what's going on there, as BIDSonym doesn't have any functionality to do these things. Could you maybe share the json files of one person with me via an encrypted transfer or could we have a short call? I'll then update the situation here accordingly.

[hcp4715]

@PeerHerholz sent you the json files via email.

[PeerHerholz]

@hcp4715 I think I've found the problem. Could you try again after pulling either the latest or v0.0.2?

[hcp4715]

Hi, @PeerHerholz, thanks so much!! I tested the latest version, there was no error, but still, some json files were messed up. I've sent them to you via email.