search

nipreps / fmriprep

fMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse fMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.
https://fmriprep.org
Apache License 2.0
638 stars 295 forks source link

`collect_derivatives` uses incorrect "from" entity while searching for pre-calculated boldref -> fmap coregistrations (orig vs boldref) #3368

Closed psadil closed 2 months ago

psadil commented 2 months ago

What happened?

fMRIPrep generates rigid xfms between the boldref and the fieldmap that have names like sub-{subject}_ses-{session}_task-{task}_run-{run}_from-boldref_to-{fieldmap_id}_mode-image_xfm.txt.

I am trying to supply precalculated xfms with the --derivatives flag, but it doesn't seem like they are being picked up.

What command did you use?

docker run \
  --init \
  --rm \
  -it \
  -v $PWD:$PWD \
  nipreps/fmriprep:24.1.0 \
  --dummy-scans 15 \
  -w $PWD/work2 \
  --fs-license-file $PWD/license \
  --derivatives boldref2fmap=$PWD/sourcedata/boldref2fmap  \
  --notrack  \
  $PWD/rawdata $PWD/derivatives2/fmriprep participant

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
2024-09-27 22:38:47,401 [ WARNING] WARNING: SOCKS support in urllib3 requires the installation of optional dependencies: specifically, PySocks.  For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies
bids-validator@1.14.10
(node:10) Warning: Closing directory handle on garbage collection
(Use `node --trace-warnings ...` to show where the warning was created)
    1: [WARN] The recommended file /README is very small. Please consider expanding it with additional information about the dataset. (code: 213 - README_FILE_SMALL)
        ./README

    Please visit https://neurostars.org/search?q=README_FILE_SMALL for existing conversations about this issue.

        Summary:                  Available Tasks:                     Available Modalities: 
        24 Files, 771.34MB        cuff                                 MRI                   
        1 - Subject               rest                                                       
        1 - Session               TODO: full task name for cuff                              
                                  TODO: full task name for rest                              

    If you have any questions, please post on https://neurostars.org/tags/bids.
2024-09-27 22:38:57,903 [ WARNING] WARNING: SOCKS support in urllib3 requires the installation of optional dependencies: specifically, PySocks.  For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies
2024-09-27 22:38:58,126 [ WARNING] WARNING: Niworkflows will be deprecating reporting in favor of a standalone library "nireports".
2024-09-27 22:38:58,251 [ WARNING] WARNING: Niworkflows will be deprecating visualizations in favor of a standalone library "nireports".
240927-22:38:59,171 nipype.workflow IMPORTANT:
     Running fMRIPrep version 24.1.0
[...]

What version of fMRIPrep are you running?

24.1.0

How are you running fMRIPrep?

Docker

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

FreeSurfer

Please copy and paste any relevant log output.

240927-22:38:59,200 nipype.workflow IMPORTANT:
     Building fMRIPrep's workflow:
           * BIDS dataset path: /Users/psadil/Desktop/nstravel2/rawdata.
           * Participant list: ['travel2'].
           * Run identifier: 20240927-223847_deb204bf-a439-41a5-a8ed-4dda4f38d9b8.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Searching for derivatives: [PosixPath('/Users/psadil/Desktop/nstravel2/sourcedata/boldref2fmap')].
           * Pre-run FreeSurfer's SUBJECTS_DIR: /Users/psadil/Desktop/nstravel2/derivatives2/fmriprep/sourcedata/freesurfer.
2024-09-27 22:38:59,200 [IMPORTANT] Building fMRIPrep's workflow:
           * BIDS dataset path: /Users/psadil/Desktop/nstravel2/rawdata.
           * Participant list: ['travel2'].
           * Run identifier: 20240927-223847_deb204bf-a439-41a5-a8ed-4dda4f38d9b8.
           * Output spaces: MNI152NLin2009cAsym:res-native.
           * Searching for derivatives: [PosixPath('/Users/psadil/Desktop/nstravel2/sourcedata/boldref2fmap')].
           * Pre-run FreeSurfer's SUBJECTS_DIR: /Users/psadil/Desktop/nstravel2/derivatives2/fmriprep/sourcedata/freesurfer.

Additional information / screenshots

I guess the issue is that the searched for pattern uses "from": "orig",?

As in, below, I would have expected "boldref" to work but not "orig" 

from pathlib import Path
from fmriprep.utils import bids
import tempfile
import json

entities = {"subject": "0", "task": "rest"}

for from_id in ["boldref", "orig"]:
    with tempfile.TemporaryDirectory() as tmpd_:
        tmpd = Path(tmpd_) / "boldref2fmap"
        funcd = tmpd / f"sub-{entities['subject']}" / "func"
        funcd.mkdir(parents=True)
        stem = f"sub-{entities['subject']}_task-{entities['task']}_from-{from_id}_to-auto00000_mode-image_xfm"
        (funcd / f"{stem}").with_suffix(".txt").write_text("placeholder")
        (funcd / f"{stem}").with_suffix(".json").write_text(
            json.dumps(entities)
        )
        derivs = bids.collect_derivatives(
            derivatives_dir=tmpd,
            entities=entities,
            fieldmap_id=None,
        )
        print(f"looking for from-{from_id}")
        print(derivs)

looking for from-boldref
defaultdict(<class 'list'>, {})
looking for from-orig
defaultdict(<class 'list'>, {'boldref2fmap': '/var/folders/v_/kcpb096s1m3_37ctfd2sp2xm0000gn/T/tmp58nmly9z/boldref2fmap/sub-0/func/sub-0_task-rest_from-orig_to-auto00000_mode-image_xfm.txt'})

FWIW, I'm trying to supply these xfms as a way to troubleshoot the issue described here: https://neurostars.org/t/poor-fieldmap-functional-registration-in-fmriprep-highly-variable-step/30397, which I mention shamelessly to get more eyes on it ;)

psadil commented 2 months ago

Unless, perhaps that field aims to pick up a different transformation -- not the boldref-auto##### generated during typical use of pepolar with fmriprep?

effigies commented 2 months ago

Yes, I think you're right. And I think the boldref2anat also needs to be updated.

effigies commented 2 months ago

Would you mind opening a PR against the maint/24.1.x branch?

psadil commented 2 months ago

Yes, can do. Just to make sure I understand how this is supposed to work, is the idea that when I run a command like

fmriprep --derivatives boldref2fmap=sourcedata/boldref2fmap [...] "${INPUT}" "${OUTPUT}" participant

where sourcedata/boldref2fmap looks like

sourcedata
└── boldref2fmap
   └── sub-travel2
      └── ses-NS
         └── func
            ├── sub-travel2_ses-NS_task-rest_run-01_from-boldref_to-auto00000_mode-image_xfm.json
            └── sub-travel2_ses-NS_task-rest_run-01_from-boldref_to-auto00000_mode-image_xfm.txt

then that xfm and json will be copied into ${OUTPUT} and used by the relevant nodes?

effigies commented 2 months ago

Not necessarily that it will be copied into the output, but that its generation will be skipped. I do not think we are 100% consistent on whether a precomputed file will be copied to output or not.

psadil commented 2 months ago

I think there might be a couple of other issues

Should I break those into separate issues and pull requests, or just do them together?

effigies commented 2 months ago

Feel free to fix together.