KeyError: 'DatasetType' on singularity run

[arovai]

Summary

singularity run produces "KeyError: 'DatasetType'" error

Additional details

• xcp_d version: 0.6.0
• Docker version: n/a
• Singularity version: 1.1.5-2.e17 (also tried on 3.6.4)

Singularity image file build like this:

singularity build pennlinc.xcp_d.0.6.0.sif docker://pennlinc/xcp_d:0.6.0

Build process ended without error. "Exact" command run:

singularity run -B /path/to/bids_folder:/data pennlinc.xcp_d.0.6.0.sif /data /data/derivatives participant --participant_label 01

Complete output:

240129-14:08:43,895 nipype.workflow IMPORTANT:
     Running xcp_d version 0.6.0:
    * fMRI directory path: /data/rawdata.
    * Participant list: ['01'].
    * Run identifier: 20240129-140829_dacc79c4-7a5a-4984-ac8a-c5ce09925b7b.

Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/cli/run.py", line 977, in build_workflow
    retval["workflow"] = init_xcpd_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/workflows/base.py", line 196, in init_xcpd_wf
    write_dataset_description(fmri_dir, os.path.join(output_dir, "xcp_d"))
  File "/usr/local/miniconda/lib/python3.8/site-packages/xcp_d/utils/bids.py", line 676, in write_dataset_description
    assert dset_desc["DatasetType"] == "derivative"
KeyError: 'DatasetType'

[tsalo]

What kind of input data are you running this on? Can you share the dataset_description.json file for your derivative dataset?

[arovai]

I have fmriprep derivatives but in "derivatives/fmriprep". Does xcp_d creates its own derivative folder ?

[tsalo]

XCP-D creates its own folder in dset/derivatives, but you need to pass in the path to the fMRIPrep derivatives to the XCP-D call, rather than the path to the raw BIDS dataset. So essentially this:

singularity run \
    -B /path/to/bids_folder:/data pennlinc.xcp_d.0.6.0.sif \
    /data/derivatives/fmriprep \
    /data/derivatives \
    participant --participant_label 01

[arovai]

Ok thanks, I though BIDS app should always take as a first argument the bids rawdata folder and not a derivative folder.

[tsalo]

That's a fair point, but the issue is that users could have fMRIPrep derivatives from multiple versions in their derivatives folder, with different folder names. It would take quite a bit of effort for XCP-D to be able to include logic to select the best derivatives from multiple options, so it's just easier (at least for now) to pass the specific derivative dataset directly.

Also, from a BIDS perspective, derivatives are also BIDS datasets, so it would make sense to pass in the BIDS dataset that will be directly ingested by the BIDS app. In XCP-D's case, the ingested dataset is the derivative dataset, not the raw dataset.

[arovai]

Sure, that make total sense given that XCP-D only need data from fmriprep. It's just that if data not used by fmriprep (and thus absent from fmriprep derivatives) happen to be useful to XCP-D (in future dev?), then it will need to get access to the rawdata. I am thinking of physiological recordings for instance (typically in rawdata but not in fmriprep) that might be useful for denoising (or maybe other things?). In the case of a BIDS app I am working on (CVRmap), it takes an optional --fmriprep_dir argument to specify the fmriprep derivatives one wish to use. If omitted, it looks into derivatives/fmriprep (which is not perfect as your comment explains!).

[tsalo]

I have been thinking of something similar (namely a --derivatives argument, based on fMRIPrep's recent enhancements), but, at least for now, XCP-D does accept a custom-confounds parameter, which lets you feed in whatever extra confounds you might want to regress out of your data. Physio confounds that have the same sampling rate as the BOLD data (e.g., RVT, RV) would work there.