search

HBClab / SoftwareIssues

Issues when running primarily fmriprep/mriqc/xcpengine (primarily on argon)
0 stars 0 forks source link

CREST fmriprep: another recon-all error #2

Open jdkent opened 4 years ago

jdkent commented 4 years ago

Script/Job File

#!/bin/bash

#$ -pe smp 16
#$ -q UI
#$ -m bea
#$ -M adriana-rivera-dompenciel@uiowa.edu
#$ -o /Shared/vosslabhpc/Projects/CREST/code/fmriprep/out/
#$ -e /Shared/vosslabhpc/Projects/CREST/code/fmriprep/err/
OMP_NUM_THREADS=10
singularity run -H ${HOME}/singularity_home -B /Shared/vosslabhpc:/mnt \
/Shared/vosslabhpc/UniversalSoftware/SingularityContainers/fmriprep-1.2.1.simg \
/mnt/Projects/CREST/ /mnt/Projects/CREST/derivatives \
participant --participant_label BETTER120044 \
-w /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep \
--write-graph --mem_mb 35000 --omp-nthreads 10 --nthreads 16 --use-aroma \
--output-space template \
--template MNI152NLin2009cAsym \
--fs-license-file /mnt/UniversalSoftware/freesurfer_license.txt

Error Message

200308-19:04:37,644 nipype.workflow WARNING:
     [Node] Error on "fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.segs_to_native_aseg.tonative" (/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative)
200308-19:04:38,854 nipype.workflow ERROR:
     Node tonative failed to run on host argon-itf-cf49-34.hpc.
200308-19:04:38,859 nipype.workflow ERROR:
     Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120044/log/20200308-100511_d66a5daa-772a-4968-b8d4-99daf24ea0a9/crash-20200308-190438-ariveradompenciel-tonative-78122109-9cd6-4992-a632-81babfdc0001.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/freesurfer/base.py", line 144, in run
    return super(FSCommand, self).run(**inputs)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 522, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 1038, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 975, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
mri_label2vol --regheader /mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/aseg.mgz --seg /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg.mgz --temp /mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/rawavg.mgz --o /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg_vol.nii.gz
Standard output:
Number of labels: 0
Annot File:      (null)
Template Volume: /mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/rawavg.mgz
Outut Volume: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg_vol.nii.gz
Registration File: (null)
Fill Threshold: 0
Label Vox Vol:  1
ProjType:       (null)
ProjTypeId:     0
ProjStart:      0
ProjStop:       0
ProjDelta:      0.1
Subject:  (null)
Hemi:     (null)
UseNewASeg2Vol:  0
DoLabelStatVol  0
LabelCodeOffset  0
setenv SUBJECTS_DIR /opt/freesurfer/subjects
$Id: mri_label2vol.c,v 1.46 2014/12/08 21:11:54 greve Exp $
Template RAS-to-Vox: --------
-1.00000  -0.00000  -0.00000   98.00000;
-0.00000  -0.00000  -1.00000   128.00000;
-0.00000   1.00000  -0.00000   128.00000;
-0.00000  -0.00000  -0.00000   1.00000;
Template Voxel Volume: 1
nHits Thresh: 0
Computing registration based on header
RegMat: --------
-1.00000   0.00000   0.00000   0.00000;
 0.00000   0.00000   1.00000   0.00000;
 0.00000  -1.00000   0.00000   0.00000;
 0.00000   0.00000   0.00000   1.00000;
Label RAS-to-Vox: --------
 1.00000   0.00000   0.00000   98.00000;
 0.00000   1.00000   0.00000   128.00000;
 0.00000   0.00000   1.00000   128.00000;
 0.00000   0.00000   0.00000   1.00000;
ERROR: loading aseg /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg.mgz
Standard error:
mghRead(/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/segs_to_native_aseg/tonative/aseg.mgz): could not read 262144 bytes at slice 151
Return code: 1

Potential Solution

delete the working directory and run again: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf

jdkent commented 4 years ago

Error when ran again:

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/fmriprep", line 11, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.6/site-packages/fmriprep/cli/run.py", line 303, in main
    fmriprep_wf.write_graph(graph2use="colored", format='svg', simple_form=True)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/workflows.py", line 430, in write_graph
    outfname = format_dot(dotfilename, format=format)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 1390, in format_dot
    formatted_dot, _ = _run_dot(dotfilename, format_ext=format)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/utils.py", line 1407, in _run_dot
    resource_monitor=False).run()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 522, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 1038, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/core.py", line 975, in raise_exception
    ).format(**runtime.dictcopy()))
RuntimeError: Command:
dot -Tsvg -o"/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.svg" "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot"
Standard output:

Standard error:
Warning: syntax ambiguity - badly delimited number '120065_' in line 2179 of /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot splits into two tokens
Error: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph.dot: syntax error in line 2179 near '120065'
Return code: 1

Potential Solution

delete the graph* files in the working directory for fmriprep: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/graph*

(follow up, this works to fix this particular error)

jdkent commented 4 years ago

Next step was deleting the freesurfer directory entirely for the participant, but this resulted in another error:

200315-11:58:00,540 nipype.workflow WARNING:
     Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

    Node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm
    Interface: fmriprep.interfaces.freesurfer.PatchedRobustRegister
    Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))

traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:00,593 nipype.workflow WARNING:
     Error while checking node hash, forcing re-run. Although this error may not prevent the workflow from running, it could indicate a major problem. Please report a new issue at https://github.com/nipy/nipype/issues adding the following information:

    Node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern
    Interface: fmriprep.interfaces.freesurfer.FSInjectBrainExtracted
    Traceback:
Traceback (most recent call last):

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/base.py", line 338, in _local_hash_check
    cached, updated = self.procs[jobid].is_cached()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )

  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )

  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))

traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:02,686 nipype.workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern" in "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/skull_strip_extern".
200315-11:58:02,793 nipype.workflow INFO:
     [Node] Setting-up "fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm" in "/nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf/surface_recon_wf/fsnative_2_t1_xfm".
200315-11:58:02,796 nipype.workflow ERROR:
     Node skull_strip_extern failed to run on host argon-itf-cf49-34.hpc.
200315-11:58:02,829 nipype.workflow ERROR:
     Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120044/log/20200315-115448_fb93b1e9-95ad-4522-8453-1fcd9f8aaff3/crash-20200315-115802-ariveradompenciel-skull_strip_extern-c2044b36-c4c2-46c3-9c9f-d7c3f3e924c9.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:05,572 nipype.workflow ERROR:
     Node fsnative_2_t1_xfm failed to run on host argon-itf-cf49-34.hpc.
200315-11:58:05,573 nipype.workflow ERROR:
     Saving crash info to /mnt/Projects/CREST/derivatives/fmriprep/sub-BETTER120044/log/20200315-115448_fb93b1e9-95ad-4522-8453-1fcd9f8aaff3/crash-20200315-115805-ariveradompenciel-fsnative_2_t1_xfm-6adebb56-12c4-4213-a796-3543734cac32.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
    result['result'] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 408, in run
    cached, updated = self.is_cached()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 294, in is_cached
    hashed_inputs, hashvalue = self._get_hashval()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 488, in _get_hashval
    self._get_inputs()
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/pipeline/engine/nodes.py", line 515, in _get_inputs
    results = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 668, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/utils/filemanip.py", line 659, in loadpkl
    unpkl = pickle.load(pkl_file)
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1440, in __setstate__
    self.trait_set( trait_change_notify = trait_change_notify, **state )
  File "/usr/local/miniconda/lib/python3.6/site-packages/traits/has_traits.py", line 1543, in trait_set
    setattr( self, name, value )
  File "/usr/local/miniconda/lib/python3.6/site-packages/nipype/interfaces/base/traits_extension.py", line 112, in validate
    self.info_text, value))
traits.trait_errors.TraitError: The trait 'T1' of a ReconAllOutputSpec instance is an existing file name, but the path  '/mnt/Projects/CREST/derivatives/freesurfer/sub-BETTER120044/mri/T1.mgz' does not exist.

200315-11:58:30,708 nipype.workflow ERROR:
     could not run node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.skull_strip_extern
200315-11:58:30,712 nipype.workflow ERROR:
     could not run node: fmriprep_wf.single_subject_BETTER120044_wf.anat_preproc_wf.surface_recon_wf.fsnative_2_t1_xfm

Potential Solution

Need to delete the relevant working directory as well: /nfsscratch/Users/ariveradompenciel/work/CRESTfmriprep/fmriprep_wf/single_subject_BETTER120044_wf/anat_preproc_wf