[ENH]: DICOM to NIFTI BIDS conversion with Heudiconv

[alexpron]

Adding datalad support within shanoir2bids.py script relying on heudiconv software.

Tried to reproduce as far as possible the current usage of the shanoir2bids.py script from configuration.json file

• DICOM download only from Shanoir instance (regression)
• removed automri format output (regression)
• BIDS mapping generated from the result of the solr search to handle case where regex like terms were passed into .json file
• BIDS conversion is performed at the subject level (session if longitudinal)
• Modified a bit dcm2niix options so that it matches the --dcmconfig option .json file (keys are attributes to DCM2NIIX nipype class)
• DICOM to NIFTI BIDS conversion, both DICOM and NIFTI are stored into datalad datasets (sourcedata, rawdata)

TODO:

• [x] check bids metadata generated
• [x] add studyname in dl_dir using locator ?
• [x] better handle temporary directory
• [x] add conversion directly from zip files
• [x] write Heudiconv heuristic to handle conversion into automri format
• [x] define what to do when chosen outputs are not BIDS compliant (e.g. sub-\d\d--\d\d)

[alexpron]

some bids compliance errors with default dcm2niix options used:

[alexpron]

• bids compliance problem was solved using -ba option of dcm2niix if -ba should not be applyied check field by field
• bids validator was run on the obtained files and current filenames used should be modified in the configuration file
• some characters are forbidden in subject_id such as - hence XX-XX is not a BIDS compliant name
• in order to keep this name use the non bids formatting and manually recreats ontology within the heuristic file ?

[alexpron]

tar or tar.gz files only are correctly handled by heudiconv (.zip files currently not supported)

[alexpron]

• [x] include debug mode with directory kept i.e. switch to manually specified directories and manual delete if required
• [x] remove commit 5064edb
• [x] remove {subject_session_dir} prefix in case of automri heuristic
• [x] remove {subject_session_prefix} in case of automri heuristic
• [x] if not working implement simple BIDS to automri mappping

Optional

to be done in a dedicated PR

• [ ] solr query enhancement : concatenate all queries into a single one

[alexpron]

• include debug mode with directory kept i.e. switch to manually specified directories and manual delete if required by default temporary directory are deleted after context execution. The delete keyword allowing to keep the directory is only available in python 3.12 which is too restrictive currently Hence temporary directory mechanism will be replaced by directories and conditional manual deletion

  See https://docs.python.org/3.12/library/tempfile.html#tempfile.TemporaryDirectory and https://docs.python.org/3.11/library/tempfile.html#tempfile.TemporaryDirectory

[alexpron]

An heuristic covering automri conversion was included into few commits. However it was decided to focus on BIDS conversion only. An script belonging to a different repo (bids_to_automri) was written as well as a PR to evaluate the time needed to go from automri to BIDS ontology as the new organisation of the automri code.

[alexpron]

request optimisation has been started but not implemented in the end as it would require to rethink the whole configuration file usage (e.g. using a proper heuristic for each study instead of direct mapping)

[alexpron]

• added basic check of BIDS path generated relying on the configuration file (bids_validator)
• overwrite mode was set to True (data are overwrite and then datalad handles their integration or not if they exist already)
• linting and improving checks

[alexpron]

• [ ] write documentation about shanoir2bids_heudiconv of the README file
• [ ] add deprecation warning for shanoir2bids ?

[quentinduche]

datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false ls-files --stage -z -- .heudiconv' failed with exitcode 128

[alexpron]

Clean Installation instructions For Debug

conda create -n test python=3.11
conda activate test 
conda install python-dotenv
conda install -c conda-forge heudiconv pybids datalad git-annex=*=alldep*