search

nipreps / fmriprep

fMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse fMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.
https://fmriprep.org
Apache License 2.0
638 stars 296 forks source link

short header read with BBR (v1.5.3) #1920

Open DVSneuro opened 4 years ago

DVSneuro commented 4 years ago

Hi -- I'm getting the error below when running FMRIPREP. I can't consistently reproduce it, but it happened on about 9 out of 50 participants. Any ideas? Thanks and Happy New Year!

Node: fmriprep_wf.single_subject_112_wf.func_preproc_task_trust_run_04_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr
Working directory: /scratch/fmriprep_wf/single_subject_112_wf/func_preproc_task_trust_run_04_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr

Node inputs:

angle_rep = <undefined>
apply_isoxfm = <undefined>
apply_xfm = <undefined>
args = <undefined>
bbrslope = <undefined>
bbrtype = <undefined>
bgvalue = <undefined>
bins = <undefined>
coarse_search = <undefined>
compress_report = auto
cost = <undefined>
cost_func = bbr
datatype = <undefined>
display_init = <undefined>
dof = 6
echospacing = <undefined>
environ = {'FSLOUTPUTTYPE': 'NIFTI_GZ'}
fieldmap = <undefined>
fieldmapmask = <undefined>
fine_search = <undefined>
force_scaling = <undefined>
in_file = <undefined>
in_matrix_file = <undefined>
in_weight = <undefined>
interp = <undefined>
min_sampling = <undefined>
no_clamp = <undefined>
no_resample = <undefined>
no_resample_blur = <undefined>
no_search = <undefined>
out_file = <undefined>
out_log = <undefined>
out_matrix_file = <undefined>
out_report = report.svg
output_type = NIFTI_GZ
padding_size = <undefined>
pedir = <undefined>
ref_weight = <undefined>
reference = <undefined>
rigid2D = <undefined>
save_log = <undefined>
schedule = /usr/share/fsl/5.0/etc/flirtsch/bbr.sch
searchr_x = <undefined>
searchr_y = <undefined>
searchr_z = <undefined>
sinc_width = <undefined>
sinc_window = <undefined>
uses_qform = <undefined>
verbose = <undefined>
wm_seg = <undefined>
wmcoords = <undefined>
wmnorms = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 479, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 585, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 678, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 382, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 767, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 704, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
flirt -in /scratch/fmriprep_wf/single_subject_112_wf/func_preproc_task_trust_run_04_wf/sdc_estimate_wf/sdc_unwarp_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz -ref /scratch/fmriprep_wf/single_subject_112_wf/anat_preproc_wf/brain_extraction_wf/copy_xform/sub-112_T1w_corrected_masked_xform.nii.gz -out uni_xform_masked_flirt.nii.gz -omat uni_xform_masked_flirt.mat -searchcost bbr -dof 6 -init /scratch/fmriprep_wf/single_subject_112_wf/func_preproc_task_trust_run_04_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.mat -schedule /usr/share/fsl/5.0/etc/flirtsch/bbr.sch -wmseg /scratch/fmriprep_wf/single_subject_112_wf/anat_preproc_wf/t1w_dseg/sub-112_T1w_corrected_masked_xform_seg_wm.nii.gz
Standard output:

Standard error:
** ERROR (nifti_image_read): short header read '/scratch/fmriprep_wf/single_subject_112_wf/anat_preproc_wf/t1w_dseg/sub-112_T1w_corrected_masked_xform_seg_wm.nii.gz'
** ERROR: nifti_image_open(/scratch/fmriprep_wf/single_subject_112_wf/anat_preproc_wf/t1w_dseg/sub-112_T1w_corrected_masked_xform_seg_wm): bad header info
ERROR: failed to open file /scratch/fmriprep_wf/single_subject_112_wf/anat_preproc_wf/t1w_dseg/sub-112_T1w_corrected_masked_xform_seg_wm
Image Exception : #22 :: ERROR: Could not open image /scratch/fmriprep_wf/single_subject_112_wf/anat_preproc_wf/t1w_dseg/sub-112_T1w_corrected_masked_xform_seg_wm
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
Aborted (core dumped)
Return code: 134
effigies commented 4 years ago

Can you look at the file/scratch/fmriprep_wf/single_subject_112_wf/anat_preproc_wf/t1w_dseg/sub-112_T1w_corrected_masked_xform_seg_wm.nii.gz? Assuming it's not just a truncated file, perhaps trying to load it in nibabel or some other tool (fslhd, mri_info) would be more informative?

DVSneuro commented 4 years ago

Thanks. I ended up having to delete my scratch folder to reclaim some disk space, so don't have that anymore. But, I did run another dataset through v1.5.4 recently and didn't encounter any issues. If I can reproduce it, I'll share more information. It seems a little strange that the output file is potentially truncated at first and then it's fine after deleting scratch and re-running.

effigies commented 4 years ago

Is it possible that you're hitting filesystem limits or quotas? That could explain a truncated file.

DVSneuro commented 4 years ago

It's possible we were running into disk space issues when running these subjects, but I don't think I saw any other errors that would be consistent with that (we've definitely filled up our disk before). I'll keep that in mind and report back if I see it again. Thanks!

mgxd commented 4 years ago

@DVSneuro feel free to reopen this if you encounter the issue again

octomike commented 4 years ago

We just encountered the same error:

Node: fmriprep_wf.single_subject_01_wf.func_preproc_task_run04_wf.bold_reg_wf.fsl_bbr_wf.flt_bbr
Working directory: /home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_run04_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr

Node inputs:

angle_rep = <undefined>
apply_isoxfm = <undefined>
apply_xfm = <undefined>
args = <undefined>
bbrslope = <undefined>
bbrtype = <undefined>
bgvalue = <undefined>
bins = <undefined>
coarse_search = <undefined>
compress_report = auto
cost = <undefined>
cost_func = bbr
datatype = <undefined>
display_init = <undefined>
dof = 6
echospacing = <undefined>
environ = {'FSLOUTPUTTYPE': 'NIFTI_GZ'}
fieldmap = <undefined>
fieldmapmask = <undefined>
fine_search = <undefined>
force_scaling = <undefined>
in_file = <undefined>
in_matrix_file = <undefined>
in_weight = <undefined>
interp = <undefined>
min_sampling = <undefined>
no_clamp = <undefined>
no_resample = <undefined>
no_resample_blur = <undefined>
no_search = <undefined>
out_file = <undefined>
out_log = <undefined>
out_matrix_file = <undefined>
out_report = report.svg
output_type = NIFTI_GZ
padding_size = <undefined>
pedir = <undefined>
ref_weight = <undefined>
reference = <undefined>
rigid2D = <undefined>
save_log = <undefined>
schedule = /usr/share/fsl/5.0/etc/flirtsch/bbr.sch
searchr_x = <undefined>
searchr_y = <undefined>
searchr_z = <undefined>
sinc_width = <undefined>
sinc_window = <undefined>
uses_qform = <undefined>
verbose = <undefined>
wm_seg = <undefined>
wmcoords = <undefined>
wmnorms = <undefined>

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 397, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 792, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 723, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
flirt -in /home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_run04_wf/bold_reference_wf/enhance_and_skullstrip_bold_wf/apply_mask/uni_xform_masked.nii.gz -ref /home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/brain_extraction_wf/copy_xform/sub-01_T1w_corrected_masked_xform.nii.gz -out uni_xform_masked_flirt.nii.gz -omat uni_xform_masked_flirt.mat -searchcost bbr -dof 6 -init /home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/func_preproc_task_run04_wf/bold_reg_wf/fsl_bbr_wf/flt_bbr_init/uni_xform_masked_flirt.mat -schedule /usr/share/fsl/5.0/etc/flirtsch/bbr.sch -wmseg /home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/t1w_dseg/sub-01_T1w_corrected_masked_xform_seg_wm.nii.gz
Standard output:

Standard error:
** ERROR (nifti_image_read): short header read '/home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/t1w_dseg/sub-01_T1w_corrected_masked_xform_seg_wm.nii.gz'
** ERROR: nifti_image_open(/home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/t1w_dseg/sub-01_T1w_corrected_masked_xform_seg_wm): bad header info
ERROR: failed to open file /home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/t1w_dseg/sub-01_T1w_corrected_masked_xform_seg_wm
Image Exception : #22 :: ERROR: Could not open image /home/mpib/sztuka/projects/RSA/work/fmriprep_wf/single_subject_01_wf/anat_preproc_wf/t1w_dseg/sub-01_T1w_corrected_masked_xform_seg_wm
terminate called after throwing an instance of 'RBD_COMMON::BaseException'
Aborted
Return code: 134

The referenced folders do not have a quota and there is definitely enough disk space available.

We are however, running this with singularity:

singularity run --cleanenv /data/container/fmriprep/fmriprep-20.0.7.sif \
    /home/mpib/sztuka/projects/RSA/niftii \
    /home/mpib/sztuka/projects/RSA/dervatives \
    participant --participant-label 01 --fs-no-reconall --fs-license-file ...

Note that there is no --contain so it's not the session directory limit.

Manually running the flirt command in the same container afterwards is working however, which rings all sorts of race condition alarm bells in my head..

effigies commented 4 years ago

Yeah, that does sound like a filesystem synchronization issue. Does rerunning resolve the problem?