search

nipreps / fmriprep

fMRIPrep is a robust and easy-to-use pipeline for preprocessing of diverse fMRI data. The transparent workflow dispenses of manual intervention, thereby ensuring the reproducibility of the results.
https://fmriprep.org
Apache License 2.0
639 stars 296 forks source link

Error with select_tpl workflow step in fmriprep_20.2.6 #2683

Open switt4 opened 2 years ago

switt4 commented 2 years ago

What happened?

Hi,

I posted this to neurostars, but I've not received any response, so I'll try here.

We recently downloaded fmrirep_20.2.6 and cannot get it to run due to an error early on with the select_tpl workflow step. From the traceback error it appears as if the command is expecting one value but is being given two.

The input dataset was collected on a 7T using the UMinn single-echo multiplex EPI sequence.

We do not get this error with fmriprep_20.2.5. Has something changed with template flow that we missed and need to update on our end?

Thanks,

Suzanne

What command did you use?

echo SINGULARITY_IMAGE: /project/6050199/akhanf/singularity/bids-apps/nipreps_fmriprep_20.2.6.sif
echo CMD: /scratch/switt4/Yalda/correct7T.v0.3/gradcorrect_v0.0.3 /scratch/switt4/Yalda/deriv/fmriprep_20.2.6 participant --participant_label $subj --nthreads 8 --n_cpus 8 --mem_mb 32000 --omp-nthreads 8 --use-plugin ${FMRIPREP_MULTIPROC_YAML} -w ${SLURM_TMPDIR} --fs-license-file ${FS_LICENSE_FILE} --resource-monitor --notrack --output-layout bids

What version of fMRIPrep are you running?

fmriprep_20.2.6

How are you running fMRIPrep?

Singularity

Is your data BIDS valid?

Yes

Are you reusing any previously computed results?

No

Please copy and paste any relevant log output.

Running fMRIPREP version 20.2.6:
      * BIDS dataset path: /scratch/switt4/Undergrads_fMRI/TAR/bids.
      * Participant list: ['001'].
      * Run identifier: 20211108-142242_3b3ca22e-c13c-42e4-bab7-ddf7d5a42849.
      * Output spaces: MNI152NLin2009cAsym:res-native.
      * Pre-run FreeSurfer's SUBJECTS_DIR: /scratch/switt4/Undergrads_fMRI/TAR/fmriprep/fmriprep_20.2.6/sourcedata/freesurfer.
211108-14:23:21,309 nipype.workflow INFO:
         No single-band-reference found for sub-001_task-movie_run-01_bold.nii.gz.
211108-14:23:23,127 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 0.96s.
211108-14:23:23,366 nipype.workflow INFO:
         No single-band-reference found for sub-001_task-rest_run-01_bold.nii.gz.
211108-14:23:23,780 nipype.workflow IMPORTANT:
         BOLD series will be slice-timing corrected to an offset of 1.06s.
211108-14:23:29,443 nipype.workflow INFO:
         fMRIPrep workflow graph with 576 nodes built successfully.
211108-14:23:46,999 nipype.workflow IMPORTANT:
         fMRIPrep started!
211108-14:23:48,179 nipype.workflow WARNING:
         [Node] Error on "fmriprep_wf.single_subject_001_wf.func_preproc_task_rest_run_01_wf.bold_std_trans_wf.select_tpl" (/localscratch/switt4.54165523.0/fmriprep_wf/single_subje>
211108-14:23:48,180 nipype.workflow ERROR:
         Node select_tpl.a0 failed to run on host gra1237.
211108-14:23:48,203 nipype.workflow ERROR:
         Saving crash info to /scratch/switt4/Undergrads_fMRI/TAR/fmriprep/fmriprep_20.2.6/sub-001/log/20211108-142242_3b3ca22e-c13c-42e4-bab7-ddf7d5a42849/crash-20211108-142348-sw>
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/legacymultiproc.py", line 432, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 715, in _run_command
    result = self._interface.run(cwd=outdir, ignore_exception=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/support.py", line 90, in __exit__
    for k, v in self._resmon.stop():
ValueError: too many values to unpack (expected 2)

Additional information / screenshots

We on running on a HPC system that uses SLURM, so apologies for the somewhat convoluted command.

mgxd commented 2 years ago

hi @switt4, sorry for the delay - can you post the contents of the crashfile located at /scratch/switt4/Undergrads_fMRI/TAR/fmriprep/fmriprep_20.2.6/sub-001/log/20211108-142242_3b3ca22e-c13c-42e4-bab7-ddf7d5a42849/crash-20211108-142348-sw...

switt4 commented 2 years ago

Hi, Below is the contents of the crashfile related to the select_tpl step. There are a lot of other crash files in this log directory, so please let me know if you need something else.

Node: fmriprep_wf.single_subject_001_wf.func_preproc_task_rest_run_01_wf.bold_std_trans_wf.select_tpl
Working directory: /localscratch/switt4.54165523.0/fmriprep_wf/single_subject_001_wf/func_preproc_task_rest_run_01_wf/bold_std_trans_wf/_std_target_MNI152NLin2009cAsym.resnative/select_tpl

Node inputs:

function_str = def _select_template(template):
    from niworkflows.utils.misc import get_template_specs
    template, specs = template
    template = template.split(':')[0]  # Drop any cohort modifier if present
    specs = specs.copy()
    specs['suffix'] = specs.get('suffix', 'T1w')

    # Sanitize resolution
    res = specs.pop('res', None) or specs.pop('resolution', None) or 'native'
    if res != 'native':
        specs['resolution'] = res
        return get_template_specs(template, template_spec=specs)[0]

    # Map nonstandard resolutions to existing resolutions
    specs['resolution'] = 2
    try:
        out = get_template_specs(template, template_spec=specs)
    except RuntimeError:
        specs['resolution'] = 1
        out = get_template_specs(template, template_spec=specs)

    return out[0]

template = ('MNI152NLin2009cAsym', {'res': 'native'})

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/legacymultiproc.py", line 432, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 715, in _run_command
    result = self._interface.run(cwd=outdir, ignore_exception=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/support.py", line 90, in __exit__
    for k, v in self._resmon.stop():
ValueError: too many values to unpack (expected 2)

When creating this crashfile, the results file corresponding
to the node could not be found.
effigies commented 2 years ago

Looks like a bug in the resource monitor. I'd suggest removing --resource-monitor from your CLI and retrying.

switt4 commented 2 years ago

I'll give that a go later this week. Our CLI for fMRIPrep is fixed for the HCP system I am running on, so I'll need to create a new test one.

Thanks!

switt4 commented 2 years ago

I just wanted to post an update that removing --resource-monitor from the CLI cleared the error and crash I was getting related to the select_tpl step.

nikhil153 commented 2 years ago

Similar issue with fmriprep_20.2.7 which also goes away with removing of --resource-monitor from the CLI. However, in future I would like to use --resource-monitor. So I thought I would add it here.

Several crash files log this error:

Crash file in: output/fmriprep/logs/crash-20220218-175632-nikhil-fsdir_run_20220218_175403_305f94ca_22b3_4da7_8001_1947608913a2-2d4563fa-6d2a-4720-9a5b-0539232c31b3.txt

    Node: fmriprep_wf.fsdir_run_20220218_175403_305f94ca_22b3_4da7_8001_1947608913a2
    Working directory: /work/fmriprep_wf/fsdir_run_20220218_175403_305f94ca_22b3_4da7_8001_1947608913a2
    Node inputs:
    derivatives = /output
    freesurfer_home = /opt/freesurfer
    overwrite_fsaverage = False
    spaces = [‘fsnative’, ‘fsaverage5’]
    subjects_dir = /fsdir
    Traceback (most recent call last):
    File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py”, line 344, in _send_procs_to_workers
    self.procs[jobid].run(updatehash=updatehash)
    File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 521, in run
    result = self._run_interface(execute=True)
    File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 639, in _run_interface
    return self._run_command(execute)
    File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py”, line 715, in _run_command
    result = self._interface.run(cwd=outdir, ignore_exception=True)
    File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py”, line 401, in run
    outputs = self.aggregate_outputs(runtime)
    File “/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/support.py”, line 90, in exit
    for k, v in self._resmon.stop():
    ValueError: too many values to unpack (expected 2)
    When creating this crashfile, the results file corresponding
    to the node could not be found.