search

nipreps / fmriprep

fMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse fMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.
https://fmriprep.org
Apache License 2.0
632 stars 293 forks source link

Isolated runs eliciting error on aCompCor and tCompCor (and confound.tsv files not produced) #1226

Closed ahafri closed 6 years ago

ahafri commented 6 years ago

I have run fmriprep on 7 subjects, each with 3-4 sessions, and 8 or so runs per session.

Out of the 7 subjects, two subjects have had one run each (different runs) in which there are errors for aCompCor and tCompCor, and as a result no confounds.tsv file is generated.

I had gotten this error for a couple more runs than this, but I think it was a memory issue then; when I allocated more RAM for the jobs (24GB), those runs no longer had errors. However, for two runs the aCompCor and tCompCor errors still exist (even without an "Out of Memory" error).

I am using fmriprep version 1.0.14 (May 15, 2018), via a singularity container. The singularity command: 

$CFNAPPS/singularity/2.5.1/bin/singularity run --bind /data/jet/ahafri:/data --bind $TMPDIR:/scratch /data/jet/ahafri/scripts/poldracklab_fmriprep_latest-2018-05-15-398ba12da641.img \
    /data/PROJECTS/event_encoding_model/data_BIDS \
    /data/PROJECTS/event_encoding_model/data_processed \
    participant --participant_label sub-s007 \
    --omp-nthreads 8 --nthreads 8 --write-graph \
    --work-dir /scratch \
    --output-space {T1w,template}

FMRIPrep command (within singularity): 

/usr/local/miniconda/bin/fmriprep /data/PROJECTS/event_encoding_model/data_BIDS /data/PROJECTS/event_encoding_model/data_processed participant --participant_label s007 --omp-nthreads 8 --nthreads 8 --write-graph --work-dir /scratch --output-space T1w template

Specific errors are identical and are listed below for one example run:

Node Name: fmriprep_wf.single_subject_s005_wf.func_preproc_ses_day01_task_book13_run_01_wf.bold_confounds_wf.tcompcor
File: /data/PROJECTS/event_encoding_model/data_processed/fmriprep/sub-s005/log/20180725-224233_3149ea49-b82f-42c5-b652-b6bfd417a037/crash-20180726-100319-ahafri-tcompcor-b0f8201c-a37b-41ad-b751-e8efb3a9a433.txt
Working Directory: /scratch/fmriprep_wf/single_subject_s005_wf/func_preproc_ses_day01_task_book13_run_01_wf/bold_confounds_wf/tcompcor
Inputs: 
components_file: tcompcor.tsv
header_prefix: <undefined>
high_pass_cutoff: 128.0
ignore_exception: False
ignore_initial_volumes: 0
mask_files: ['/scratch/fmriprep_wf/single_subject_s005_wf/func_preproc_ses_day01_task_book13_run_01_wf/bold_confounds_wf/tcc_tfm/highres001_N4Corrected0_rbrainmask_eroded_trans_boldmsk.nii.gz']
mask_index: <undefined>
merge_method: <undefined>
num_components: 6
percentile_threshold: 0.05
pre_filter: cosine
realigned_file: /scratch/fmriprep_wf/single_subject_s005_wf/func_preproc_ses_day01_task_book13_run_01_wf/bold_bold_trans_wf/merge/vol0000_xform-00000_merged.nii.gz
regress_poly_degree: 1
repetition_time: 2.0
save_pre_filter: True
use_regress_poly: <undefined>
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 68, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 480, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 564, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 644, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 520, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/patches.py", line 51, in _run_interface
    runtime = super(RobustTCompCor, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 517, in _run_interface
    mask_images = self._process_masks(mask_images, imgseries.get_data())
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 687, in _process_masks
    (1. - self.inputs.percentile_threshold)).astype(int))
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4116, in percentile
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/lib/function_base.py", line 3858, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/lib/function_base.py", line 4233, in _percentile
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 134, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/core/fromnumeric.py", line 57, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
Node Name: fmriprep_wf.single_subject_s005_wf.func_preproc_ses_day01_task_book13_run_01_wf.bold_confounds_wf.acompcor
File: /data/PROJECTS/event_encoding_model/data_processed/fmriprep/sub-s005/log/20180725-224233_3149ea49-b82f-42c5-b652-b6bfd417a037/crash-20180726-100319-ahafri-acompcor-f93523be-94b6-4e1c-9137-85f88d826512.txt
Working Directory: /scratch/fmriprep_wf/single_subject_s005_wf/func_preproc_ses_day01_task_book13_run_01_wf/bold_confounds_wf/acompcor
Inputs: 
components_file: acompcor.tsv
header_prefix: <undefined>
high_pass_cutoff: 128.0
ignore_exception: False
ignore_initial_volumes: 0
mask_files: ['/scratch/fmriprep_wf/single_subject_s005_wf/func_preproc_ses_day01_task_book13_run_01_wf/bold_confounds_wf/acc_tfm/highres001_N4Corrected0_masked_prob_0_tpmsum_roi_trans_boldmsk.nii.gz']
mask_index: <undefined>
merge_method: <undefined>
num_components: 6
pre_filter: cosine
realigned_file: /scratch/fmriprep_wf/single_subject_s005_wf/func_preproc_ses_day01_task_book13_run_01_wf/bold_bold_trans_wf/merge/vol0000_xform-00000_merged.nii.gz
regress_poly_degree: 1
repetition_time: 2.0
save_pre_filter: True
use_regress_poly: <undefined>
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/plugins/multiproc.py", line 68, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 480, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 564, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/pipeline/engine/nodes.py", line 644, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/interfaces/base/core.py", line 520, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/interfaces/patches.py", line 28, in _run_interface
    runtime = super(RobustACompCor, self)._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 539, in _run_interface
    self.inputs.pre_filter, degree, self.inputs.high_pass_cutoff, TR)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 1172, in compute_noise_components
    voxel_timecourses, repetition_time, period_cut)
  File "/usr/local/miniconda/lib/python3.6/site-packages/niworkflows/nipype/algorithms/confounds.py", line 1022, in cosine_filter
    betas = np.linalg.lstsq(X, data.T)[0]
  File "/usr/local/miniconda/lib/python3.6/site-packages/numpy/linalg/linalg.py", line 1965, in lstsq
    0, work, lwork, iwork, 0)
ValueError: On entry to DLALSD parameter number 4 had an illegal value
oesteban commented 6 years ago

Hi @ahafri, this looks like a memory error (potentially duplicate of #1162, #1209, etc.).

First, we just have released 1.1.3 with a bugfix for this. Please check out that new version.

If the error is still happening, please set the flag --mem-mb to a low value (lower than your actual memory ceiling). For instance, if your machine has 32GB RAM, set that to some 20GB - --mem-mb 20000.

If this is still happening, then try the flat --low-mem.

Please let us know if this worked for you.

ahafri commented 6 years ago

Hey, I tried 1.1.3 but was having Freesurfer license issues (wasn't detecting my license file) so I stuck with 1.0.14. However, the --mem-mb 20000 worked: no aCompCor or tCompCor errors!

oesteban commented 6 years ago

Thanks for the update.

I strongly recommend you to update to the latest release (1.1.4), so please open a new issue describing what you tried, and how it doesn't work for you.

Also, please check on neurostars.org for users having similar issues: https://neurostars.org/search?q=license%20tags%3Afmriprep