No preprocessed derivative for bold found

[ghost]

Dear Erin,

I am trying to get fmriprep_ciftify to run on a subject that has already been preprocessed with fmriprep. I used the following command:

docker run -ti --rm \ -v /Applications/freesurfer/license.txt:/opt/freesurfer/license.txt:ro \ -v ~/Documents/bids/Nifti:/data:ro \ -v ~/Documents/bids/out:/out \ tigrlab/fmriprep_ciftify:latest \ /data /out participant --participant_label=A00062411 \ --fs-license /opt/freesurfer/license.txt --debug \ --read-from-derivatives ~/Documents/bids/out/

And I'm getting the following error:

Running: cifti_vis_recon_all subject --ciftify-work-dir /out/ciftify sub-A00062411 No preprocessed derivative for bold not found. Skipping ciftify_subject_fmri. Please check preprocessing pipeline for errors

I wonder why ciftify is not seeing the preprocessed bold file. Below is the structure of the derivatives folder (~/Documents/bids/out/fmriprep/)

sub-A00062411 ├── anat │   ├── sub-A00062411_desc-brain_mask.json │   ├── sub-A00062411_desc-brain_mask.nii.gz │   ├── sub-A00062411_desc-preproc_T1w.json │   ├── sub-A00062411_desc-preproc_T1w.nii.gz │   ├── sub-A00062411_dseg.nii.gz │   ├── sub-A00062411_from-MNI152NLin2009cAsym_to-T1w_mode-image_xfm.h5 │   ├── sub-A00062411_from-T1w_to-MNI152NLin2009cAsym_mode-image_xfm.h5 │   ├── sub-A00062411_label-CSF_probseg.nii.gz │   ├── sub-A00062411_label-GM_probseg.nii.gz │   ├── sub-A00062411_label-WM_probseg.nii.gz │   ├── sub-A00062411_space-MNI152NLin2009cAsym_desc-brain_mask.json │   ├── sub-A00062411_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz │   ├── sub-A00062411_space-MNI152NLin2009cAsym_desc-preproc_T1w.json │   ├── sub-A00062411_space-MNI152NLin2009cAsym_desc-preproc_T1w.nii.gz │   ├── sub-A00062411_space-MNI152NLin2009cAsym_dseg.nii.gz │   ├── sub-A00062411_space-MNI152NLin2009cAsym_label-CSF_probseg.nii.gz │   ├── sub-A00062411_space-MNI152NLin2009cAsym_label-GM_probseg.nii.gz │   └── sub-A00062411_space-MNI152NLin2009cAsym_label-WM_probseg.nii.gz ├── figures │   ├── sub-A00062411_dseg.svg │   └── sub-A00062411_space-MNI152NLin2009cAsym_T1w.svg └── ses-DSA ├── anat │   └── sub-A00062411_ses-DSA_from-orig_to-T1w_mode-image_xfm.txt ├── figures │   ├── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-carpetplot_bold.svg │   ├── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-compcorvar_bold.svg │   ├── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-confoundcorr_bold.svg │   ├── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-flirtbbr_bold.svg │   ├── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-rois_bold.svg │   └── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-sdc_bold.svg └── func ├── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-confounds_regressors.json ├── sub-A00062411_ses-DSA_task-rest_acq-1400_desc-confounds_regressors.tsv ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-MNI152NLin2009cAsym_boldref.nii.gz ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-MNI152NLin2009cAsym_desc-brain_mask.json ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-MNI152NLin2009cAsym_desc-preproc_bold.json ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-T1w_boldref.nii.gz ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-T1w_desc-brain_mask.json ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-T1w_desc-brain_mask.nii.gz ├── sub-A00062411_ses-DSA_task-rest_acq-1400_space-T1w_desc-preproc_bold.json └── sub-A00062411_ses-DSA_task-rest_acq-1400_space-T1w_desc-preproc_bold.nii.gz

I've also attached the output of the command and cifti_recon_all.log cifti_recon_all.log fmriprep_ciftify output.txt

Any help is greatly appreciated! Thanks, Olena

[edickie]

Weird. Looks like pybids is not parsing the fmriprep outputs as expected...will investigate

Thanks for letting me know.

[ghost]

Thank you! Right now, I'm getting a bunch of ResourceWarnings, DeprecationWarnings, and ImportWarnings as fmriprep runs (fmriprep_ciftify log.txt), and then it hangs for hours while running the following command:

ciftify_recon_all --n_cpus 1 --ciftify-work-dir /out/ciftify --fs-subjects-dir /Users/neuroteam/Documents/bids/Nifti/derivatives/freesurfer --surf-reg MSMSulc sub-A00040573

[edickie]

So, I don't know if it's hanging..The msm algorithm does literally take hours...

The warnings are all expected (there's actually mostly issues Warning about things that will be deprecated in python 3.8 - for a lot of the dependant python packages - like numpy).

On Thu, Jun 13, 2019 at 10:46 AM OlenaInuki notifications@github.com wrote:

Thank you! Right now, I'm getting a bunch of ResourceWarnings, DeprecationWarnings, and ImportWarnings as fmriprep runs (fmriprep_ciftify log.txt https://github.com/edickie/ciftify/files/3286721/fmriprep_ciftify.log.txt), and then it hangs for hours while running the following command:

ciftify_recon_all --n_cpus 1 --ciftify-work-dir /out/ciftify --fs-subjects-dir /Users/neuroteam/Documents/bids/Nifti/derivatives/freesurfer --surf-reg MSMSulc sub-A00040573

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/edickie/ciftify/issues/113?email_source=notifications&email_token=ADEXT5RU3GQMVOIRCKFNF4LP2JMWBA5CNFSM4HWVFTE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXT56ZI#issuecomment-501735269, or mute the thread https://github.com/notifications/unsubscribe-auth/ADEXT5X7AQBLYFWWXWEBABDP2JMWBANCNFSM4HWVFTEQ .

[ghost]

Thanks for getting back to me so fast! fmriprep_ciftify just finished running cifti_vis_recon_all and exited. However, ciftify_subject_fmri does not seem to have been run, and the MNINonLinear/Results folder is empty. I'm not getting any errors about ciftify not being able to find the BOLD data, though. (cifti_recon_all.log).

I wonder if this has something to do with the fact that my preprocessed BOLD data (sub-A00040573_task-rest_space-T1w_desc-preproc_bold.nii.gz) seem to have different dimensions than the T1w image, and I'm not sure why that is.

fslinfo ~/derivatives/fmriprep/sub-A00040573/anat/sub-A00040573_desc-preproc_T1w.nii.gz data_type FLOAT32 dim1 176 dim2 256 dim3 256 dim4 1 datatype 16 pixdim1 1.000000 pixdim2 0.976562 pixdim3 0.976562 pixdim4 0.000000

fslinfo ~/derivatives/fmriprep/sub-A00040573/func/sub-A00040573_task-rest_space-T1w_desc-preproc_bold.nii.gz data_type INT16 dim1 66 dim2 90 dim3 69 dim4 404 datatype 4 pixdim1 2.000000 pixdim2 2.000000 pixdim3 2.000000 pixdim4 1.400000

Thank you again! Olena

[edickie]

It's not the voxel size, I think fMRIprep usually leaves the _bold data in the same voxel size as the original aquisition. That's expected.

It must be an error in the way that the bids_app runner is finding the preprocessed files using pybids..sorry about this.

On Thu, Jun 13, 2019 at 11:41 AM OlenaInuki notifications@github.com wrote:

Thanks for getting back to me so fast! fmriprep_ciftify just finished running cifti_vis_recon_all and exited. However, ciftify_subject_fmri does not seem to have been run, and the MNINonLinear/Results folder is empty. I'm not getting any errors about ciftify not being able to find the BOLD data, though. (cifti_recon_all.log https://github.com/edickie/ciftify/files/3286944/cifti_recon_all.log).

I wonder if this has something to do with the fact that my preprocessed BOLD data (sub-A00040573_task-rest_space-T1w_desc-preproc_bold.nii.gz) seem to have different dimensions than the T1w image, and I'm not sure why that is.

fslinfo ~/derivatives/fmriprep/sub-A00040573/anat/sub-A00040573_desc-preproc_T1w.nii.gz data_type FLOAT32 dim1 176 dim2 256 dim3 256 dim4 1 datatype 16 pixdim1 1.000000 pixdim2 0.976562 pixdim3 0.976562 pixdim4 0.000000

fslinfo ~/derivatives/fmriprep/sub-A00040573/func/sub-A00040573_task-rest_space-T1w_desc-preproc_bold.nii.gz data_type INT16 dim1 66 dim2 90 dim3 69 dim4 404 datatype 4 pixdim1 2.000000 pixdim2 2.000000 pixdim3 2.000000 pixdim4 1.400000

Thank you again! Olena

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/edickie/ciftify/issues/113?email_source=notifications&email_token=ADEXT5QX7OJHDZU36PMQJFTP2JTDFA5CNFSM4HWVFTE2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXUDSSQ#issuecomment-501758282, or mute the thread https://github.com/notifications/unsubscribe-auth/ADEXT5VALZLVKIRQTVIUB5LP2JTDFANCNFSM4HWVFTEQ .

[edickie]

Wait. I just realized the command looks off. You need the --read-from-derivatives option to point to the location within the container.. Does this work?

docker run -ti --rm -v /Applications/freesurfer/license.txt:/opt/freesurfer/license.txt:ro -v ~/Documents/bids/Nifti:/data:ro -v ~/Documents/bids/out:/out tigrlab/fmriprep_ciftify:latest /data /out participant --participant_label=A00062411 --fs-license /opt/freesurfer/license.txt --debug --read-from-derivatives /out/

[ghost]

Dear Erin,

I wanted to update you on this issue. I set the --read-from-derivatives option to point to a folder in the container, as you suggested but still wasn't able to get ciftify_subject_fmri to run.

It looks like the problem was with file naming. I wasn't getting any BIDS validator warnings when I ran fmriprep on these data, but got a warning about inconsistent file names when I ran fmriprep_ciftify. It looks like the problem was that some participants in the bids directory had somewhat different bold file names, and even though I wasn't actually running these participants, fmriprep_ciftify wouldn't run ciftify_subject_fmri because of these inconsistencies. So, I renamed all the functional files in the BIDS directory, and ciftify_subject_fmri was able to run successfully.

Thank you so much for all your help! Olena

[QuintinFrerichsWUSTL]

With command: sudo docker run -ti --rm -v /home/quintin_frerichs/data/mdma-fmri-imperial-copy:/data -v /home/quintin_frerichs/data/fmriprep-output/mdma-fmri-imperial-output/derivatives:/out -v /home/quintin_frerichs/data/ignore:/ignore tigrlab/fmriprep_ciftify:latest /data /out participant --participant_label 01 --read-from-derivatives /out --fs-license /ignore/license.txt --n_cpus 8 --resample-to-T1w32k --debug

I'm having a similar issue here, but I don't think it's anything to do with my file naming conventions/output of fmriprep having inconsistent file names. I'm just working with a single subject here, and my derivatives folder looks like this:

│   └── func │   ├── sub-01_ses-1_task-rest_run-1_desc-confounds_timeseries.json │   ├── sub-01_ses-1_task-rest_run-1_desc-confounds_timeseries.tsv │   ├── sub-01_ses-1_task-rest_run-1_from-T1w_to-scanner_mode-image_xfm.txt │   ├── sub-01_ses-1_task-rest_run-1_from-scanner_to-T1w_mode-image_xfm.txt │   ├── sub-01_ses-1_task-rest_run-1_space-MNI152NLin2009cAsym_boldref.nii.gz │   ├── sub-01_ses-1_task-rest_run-1_space-MNI152NLin2009cAsym_desc-aparcaseg_dseg.nii.gz │   ├── sub-01_ses-1_task-rest_run-1_space-MNI152NLin2009cAsym_desc-aseg_dseg.nii.gz │   ├── sub-01_ses-1_task-rest_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.json │   ├── sub-01_ses-1_task-rest_run-1_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz │   ├── sub-01_ses-1_task-rest_run-1_space-MNI152NLin2009cAsym_desc-preproc_bold.json │   ├── sub-01_ses-1_task-rest_run-1_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz │   ├── sub-01_ses-1_task-rest_run-2_desc-confounds_timeseries.json │   ├── sub-01_ses-1_task-rest_run-2_desc-confounds_timeseries.tsv │   ├── sub-01_ses-1_task-rest_run-2_from-T1w_to-scanner_mode-image_xfm.txt │   ├── sub-01_ses-1_task-rest_run-2_from-scanner_to-T1w_mode-image_xfm.txt │   ├── sub-01_ses-1_task-rest_run-2_space-MNI152NLin2009cAsym_boldref.nii.gz │   ├── sub-01_ses-1_task-rest_run-2_space-MNI152NLin2009cAsym_desc-aparcaseg_dseg.nii.gz │   ├── sub-01_ses-1_task-rest_run-2_space-MNI152NLin2009cAsym_desc-aseg_dseg.nii.gz │   ├── sub-01_ses-1_task-rest_run-2_space-MNI152NLin2009cAsym_desc-brain_mask.json │   ├── sub-01_ses-1_task-rest_run-2_space-MNI152NLin2009cAsym_desc-brain_mask.nii.gz │   ├── sub-01_ses-1_task-rest_run-2_space-MNI152NLin2009cAsym_desc-preproc_bold.json │   └── sub-01_ses-1_task-rest_run-2_space-MNI152NLin2009cAsym_desc-preproc_bold.nii.gz

Again for me, ciftify_recon_all works, but ciftify_subject_fmri doesn't:

No preprocessed derivative for bold not found. Skipping ciftify_subject_fmri. Please check preprocessing pipeline for errors No preprocessed derivative for bold not found. Skipping ciftify_subject_fmri. Please check preprocessing pipeline for errors No preprocessed derivative for bold not found. Skipping ciftify_subject_fmri. Please check preprocessing pipeline for errors No preprocessed derivative for bold not found. Skipping ciftify_subject_fmri. Please check preprocessing pipeline for errors

[edickie]

ciftify would be looking for the space-T1w outputs - so you might have not run fmriprep in a way that would output those?

[joebathelt]

I have a similar issue. Namely, I ran FreeSurfer separately to make use of multiple anatomical scans that I had for some of the participants. I'm now trying to get ciftify to pick up the FreeSurfer reconstructions and process the fMRI data. I tried running it like this:

docker run -ti --rm \
    -v $HOME/rsfMRI_language/data/bids:/data:ro \
    -v $HOME/Documents/1_Projects/rsfMRI_language/data/ciftify:/out \
    -v $HOME/rsfMRI_language/data/bids/derivatives/:/derivatives \
    -v $HOME/Documents/1_Projects/code/license.txt:/fs_license.txt \
    tigrlab/fmriprep_ciftify:latest \
    /data /out/out participant \
    --participant_label=101 \
    --read-from-derivatives=/derivatives/ \
    --fs-license=/fs_license.txt \
    --n_cpus 8

My folder structure looks like this:

bids └──  derivatives/ │ └────  freesurfer/ │ │ └────  sub-101/ .... └──  sub-101/ │ └────  anat/ │ │ ├────  sub-101_run-1_T1w.json │ │ ├────  sub-101_run-1_T1w.nii.gz │ │ ├────  sub-101_run-2_T1w.json │ │ └────  sub-101_run-2_T1w.nii.gz │ └────  func/ │ │ ├────  sub-101_task-rest_bold.json │ │ └────  sub-101_task-rest_bold.nii.gz

I get the following error message:

No preprocessed derivative for bold <BIDSFile filename='sub-101/func/sub-101_task-rest_bold.nii.gz'> not found. Skipping ciftify_subject_fmri. Please check preprocessing pipeline for errors

Do I have to run ciftify_reconall before?

Thank you in advance!

[edickie]

So the main issue is that the "--read-from-derivatives" flag will skip all calls to fmriprep (including the bold preprocessing steps).

You can actually get ciftify to call fmriprep by not using the --read-from-derivatives and setting the output directory to one directory above the freesurfer directory:

docker run -ti --rm \
    -v $HOME/rsfMRI_language/data/bids:/data:ro \
    -v $HOME/rsfMRI_language/data/bids/derivatives/:/derivatives \
    -v $HOME/Documents/1_Projects/code/license.txt:/fs_license.txt \
    tigrlab/fmriprep_ciftify:latest \
    /data /derivatives participant \
    --participant_label=101 \
    --fs-license=/fs_license.txt \
    --n_cpus 8

Alternatively, if you would like a newer version of fmriprep to run - run fmriprep independently first with the version or your choice. Just remember to add the outputs include "space-T1w" bold outputs.

[joebathelt]

Brilliant, that works. Thank you very much for the swift help!

[joebathelt]

Sorry, a follow-up issue cropped up. I now processed everything with FreeSurfer and the latest fmriprep outside of the ciftify pipeline. I'm now trying to get ciftify to run on the output from both:

docker run -ti --rm \
    -v $HOME/rsfMRI_language/data/bids:/data:ro \
    -v $HOME/rsfMRI_language/data/bids/derivatives:/derivatives \
    -v $HOME/Documents/1_Projects/code/license.txt:/fs_license.txt \
    tigrlab/fmriprep_ciftify:latest \
    /data /derivatives participant \
    --participant_label=101 \
    --read-from-derivatives=/derivatives \
    --fs-license=/fs_license.txt \

The fmriprep and freesurfer folders are in the derivatives folder:

bids └──  derivatives/ │ └────  freesurfer/ │ │ └────  sub-101/ │ └────  fmriprep/ │ │ └────  sub-101/

The outputs are all there, but ciftify seems to have some difficulties with picking them up:

cmd: ciftify_recon_all --n_cpus 1 --ciftify-work-dir /derivatives/ciftify --fs-subjects-dir /derivatives/freesurfer --surf-reg MSMSulc sub-101
 Failed with returncode 1
cmd: mri_convert /derivatives/freesurfer/sub-101/mri/T1.mgz /derivatives/ciftify/sub-101/T1w/T1w.nii.gz
 Failed with returncode 255

Is something going wrong with the mount points?

[edickie]

It's looks like the issue is that the bids data is mounted as read-only, so the derivatives (that are nested underneath the data) can't be written there. You have two options.

1) remove the "ro" from the data mount.

 docker run -ti --rm \
    -v $HOME/rsfMRI_language/data/bids:/data \
    -v $HOME/Documents/1_Projects/code/license.txt:/fs_license.txt \
    tigrlab/fmriprep_ciftify:latest \
    /data /data/derivatives participant \
    --participant_label=101 \
    --read-from-derivatives=/data/derivatives \
    --fs-license=/fs_license.txt \
    --n_cpus 8

2) or you could write the ciftify folder to another folder (completely outside the data mount.

docker run -ti --rm \
    -v $HOME/rsfMRI_language/data/bids:/data:ro \
    -v $HOME/rsfMRI_language/data/ciftify:/ciftify \
    -v $HOME/Documents/1_Projects/code/license.txt:/fs_license.txt \
    tigrlab/fmriprep_ciftify:latest \
    /data /ciftify participant \
    --participant_label=101 \
    --read-from-derivatives=/data/derivatives \
    --fs-license=/fs_license.txt \

[joebathelt]

Unfortunately, both solutions did not work. I still get the same error.

Is there a way to run ciftify interactively without triggering the main script? I could then check the folder structure inside the container.

[joebathelt]

Mystery solved: After digging around in the filesystem of the image by running docker interactively and using the --entrypoint /bin/bash flag, I found out that there was an issue with my FreeSurfer license file. My computer had offloaded it to the cloud (thanks, Apple) and the mounted file was, therefore, empty. Thank you, @edickie, for your help!

[edickie]

Glad to hear you figured this out!