search

PennLINC / qsiprep

Preprocessing and reconstruction of diffusion MRI
http://qsiprep.readthedocs.io
BSD 3-Clause "New" or "Revised" License
136 stars 54 forks source link

AMICO sort of failing #400

Open araikes opened 2 years ago

araikes commented 2 years ago

Hi @mattcieslak ,

I was running the amico_noddi recon pipeline. The NIFTI files get written but the process appears to have stalled and has been in the same spot for the last 12 hours:

220414-22:09:38,516 nipype.workflow INFO:
     [Node] Executing "plot_peaks" <qsiprep.interfaces.reports.CLIReconPeaksReport>
220414-22:11:35,633 nipype.workflow INFO:
     [Node] Finished "plot_peaks", elapsed time 116.974322s.

I got the following errors earlier in the process but haven't found any error log outside of the output as the process went (copying the full output as I have it in the terminal):

singularity run --cleanenv -B nifti:/nifti:ro -B derivatives/qsiprep-0.15.3/:/output -B /groups/adamraikes/license.txt:/license.txt /groups/adamraikes/singularity_images/qsiprep-0.15.3.sif /nifti /output/amico participant --participant_label 168 --nthreads 24 --omp-nthreads 6 --output-space T1w --output_resolution 1 --recon-only --recon-spec amico_noddi --recon-input /output/qsiprep -w /output/scratch --fs-license-file /license.txt
220414-21:45:16,604 nipype.utils WARNING:
     A newer version (1.7.1) of nipy/nipype is available. You are using 1.7.0
220414-21:45:18,82 nipype.workflow IMPORTANT:

    Running qsirecon version 0.15.3:
      * BIDS dataset path: /nifti.
      * Participant list: ['168'].
      * Run identifier: 20220414-214516_21e82c34-ae16-4188-928d-9cda0c2f6a16.

220414-21:45:18,259 nipype.workflow INFO:
     found ['/output/qsiprep/sub-168/ses-base/dwi/sub-168_ses-base_rec-dfc_space-T1w_desc-preproc_dwi.nii.gz', '/output/qsiprep/sub-168/ses-base/dwi/sub-168_ses-base_space-T1w_desc-preproc_dwi.nii.gz'] in /output/qsiprep
220414-21:45:18,260 nipype.workflow INFO:
     Found usable QSIPrep-preprocessed T1w image and mask.
220414-21:45:18,262 nipype.workflow INFO:
     Found T1w-to-template transforms
220414-21:45:18,262 nipype.workflow INFO:
     No FreeSurfer inputs available for 168
220414-21:45:18,262 nipype.workflow INFO:
     Anatomical (T1w) available for recon: {'has_qsiprep_5tt_fast': False, 'has_qsiprep_5tt_hsvs': False, 'has_freesurfer_5tt_hsvs': False, 'has_freesurfer': False, 'has_qsiprep_t1w': True, 'has_qsiprep_t1w_transforms': True}
220414-21:45:18,265 nipype.workflow INFO:
     Transforming ODF ROIs into DWI space for visual report.
220414-21:45:18,312 nipype.workflow INFO:
     Transforming ODF ROIs into DWI space for visual report.
220414-21:45:20,648 nipype.workflow IMPORTANT:
     Works derived from this qsiprep execution should include the following boilerplate:

Reconstruction was
performed using *QSIprep* 0.15.3,
which is based on *Nipype* 1.7.0
(@nipype1; @nipype2; RRID:SCR_002502).

QSIPrep-preprocessed T1w images and brain masks were used. NODDI Reconstruction

: The NODDI model (@noddi) was fit using the AMICO implementation (@amico).
A value of 1.7E-03 was used for parallel diffusivity and 3.0E-03 for isotropic
diffusivity.

Many internal operations of *qsiprep* use
*Nilearn* 0.9.0 [@nilearn, RRID:SCR_001362] and
*Dipy* 1.4.1[@dipy].
For more details of the pipeline, see [the section corresponding
to workflows in *qsiprep*'s documentation](https://qsiprep.readthedocs.io/en/latest/workflows.html "qsiprep's documentation").

### References

[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'CITATION'
220414-21:45:27,976 nipype.workflow INFO:
     [Node] Setting-up "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_space_T1w_desc_preproc_recon_wf.qsirecon_anat_wf.resample_mask" in "/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/resample_mask".
220414-21:45:28,2 nipype.workflow INFO:
     [Node] Setting-up "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf.qsirecon_anat_wf.odf_rois" in "/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/odf_rois".
220414-21:45:28,35 nipype.workflow INFO:
     [Node] Setting-up "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_space_T1w_desc_preproc_recon_wf.qsirecon_anat_wf.odf_rois" in "/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/odf_rois".
220414-21:45:30,35 nipype.workflow INFO:
     [Node] Executing "odf_rois" <nipype.interfaces.ants.resampling.ApplyTransforms>
220414-21:45:30,75 nipype.workflow INFO:
     [Node] Executing "resample_mask" <nipype.interfaces.afni.utils.Resample>
220414-21:45:30,75 nipype.workflow WARNING:
     [Node] Error on "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_space_T1w_desc_preproc_recon_wf.qsirecon_anat_wf.resample_mask" (/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/resample_mask)
exception calling callback for <Future at 0x7f96a2ee33d0 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 715, in _run_command
    result = self._interface.run(cwd=outdir, ignore_exception=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 388, in run
    self._check_mandatory_inputs()
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 275, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: Resample requires a value for input 'in_file'. For a list of required inputs, see Resample.help()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 70, in run_node
    result["result"] = node.result
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 220, in result
    return _load_resultfile(
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/utils.py", line 291, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/resample_mask/result_resample_mask.pklz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/_base.py", line 328, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
FileNotFoundError: /output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/resample_mask/result_resample_mask.pklz
220414-21:45:30,94 nipype.workflow INFO:
     [Node] Executing "odf_rois" <nipype.interfaces.ants.resampling.ApplyTransforms>
220414-21:45:30,94 nipype.workflow WARNING:
     [Node] Error on "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_space_T1w_desc_preproc_recon_wf.qsirecon_anat_wf.odf_rois" (/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/odf_rois)
exception calling callback for <Future at 0x7f96a2e8e4c0 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 715, in _run_command
    result = self._interface.run(cwd=outdir, ignore_exception=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 388, in run
    self._check_mandatory_inputs()
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 275, in _check_mandatory_inputs
    raise ValueError(msg)
ValueError: ApplyTransforms requires a value for input 'transforms'. For a list of required inputs, see ApplyTransforms.help()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 70, in run_node
    result["result"] = node.result
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 220, in result
    return _load_resultfile(
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/utils.py", line 291, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/odf_rois/result_odf_rois.pklz
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/_base.py", line 328, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
FileNotFoundError: /output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_space_T1w_desc_preproc_recon_wf/qsirecon_anat_wf/odf_rois/result_odf_rois.pklz
220414-21:45:31,861 nipype.workflow INFO:
     [Node] Setting-up "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf.fit_noddi.recon_noddi" in "/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf/fit_noddi/recon_noddi".
220414-21:45:31,916 nipype.workflow INFO:
     [Node] Executing "recon_noddi" <qsiprep.interfaces.amico.NODDI>
-> Writing scheme file to [ /output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf/fit_noddi/recon_noddi/study/subject/sub-168_ses-base_rec-dfc_space-T1w_desc-preproc_dwi.scheme ]

-> Loading data:
    * DWI signal
        - dim    = 195 x 232 x 197 x 137
        - pixdim = 1.000 x 1.000 x 1.000
    * Acquisition scheme
        - 137 samples, 3 shells
        - 13 @ b=0 , 12 @ b=500.0 , 48 @ b=1000.0 , 64 @ b=2000.0 
    * Binary mask
        - dim    = 195 x 232 x 197
        - pixdim = 1.000 x 1.000 x 1.000
        - voxels = 1506085
   [ 18.9 seconds ]

-> Preprocessing:
    * Normalizing to b0... [ min=0.00,  mean=0.45, max=17.05 ]
    * Keeping all b0 volume(s)
   [ 3.8 seconds ]
220414-21:45:54,692 nipype.interface INFO:
     Fitting NODDI Model.

-> Creating LUT for "NODDI" model:
   |\u2588\u2588\u2588\u2588\u2588\u2588\u2589                                                    | 11.7%220414-21:46:20,387 nipype.workflow INFO:
     [Node] Finished "odf_rois", elapsed time 49.863391s.
   |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 100.0%
   [ 202.6 seconds ]

-> Resampling LUT for subject "subject":
   |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 100.0%
   [ 38.3 seconds ]

-> Fitting "NODDI" model to 1506085 voxels:
   |\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 100.0%
   [ 00h 19m 19s ]

-> Saving output to "AMICO/NODDI/*":
    - configuration  [OK]
    - FIT_dir.nii.gz  [OK]
    - FIT_ICVF.nii.gz  [OK]
    - FIT_OD.nii.gz  [OK]
    - FIT_ISOVF.nii.gz  [OK]
   [ DONE ]
220414-22:09:22,368 nipype.workflow INFO:
     [Node] Finished "recon_noddi", elapsed time 1430.431711s.
220414-22:09:26,886 nipype.workflow INFO:
     [Node] Setting-up "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf.fit_noddi.convert_to_fibgz" in "/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf/fit_noddi/convert_to_fibgz".
220414-22:09:26,919 nipype.workflow INFO:
     [Node] Executing "convert_to_fibgz" <qsiprep.interfaces.converters.NODDItoFIBGZ>
220414-22:09:27,559 nipype.workflow INFO:
     Detecting Peaks
220414-22:09:35,935 nipype.workflow INFO:
     [Node] Finished "convert_to_fibgz", elapsed time 8.997706s.
220414-22:09:38,480 nipype.workflow INFO:
     [Node] Setting-up "qsirecon_wf.sub-168_amico_noddi.sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf.fit_noddi.plot_peaks" in "/output/scratch/qsirecon_wf/sub-168_amico_noddi/sub_168_ses_base_rec_dfc_space_T1w_desc_preproc_recon_wf/fit_noddi/plot_peaks".
220414-22:09:38,516 nipype.workflow INFO:
     [Node] Executing "plot_peaks" <qsiprep.interfaces.reports.CLIReconPeaksReport>
220414-22:11:35,633 nipype.workflow INFO:
     [Node] Finished "plot_peaks", elapsed time 116.974322s.
mattcieslak commented 2 years ago

Can you try re-running it with --skip-odf-reports? There's something going on with the odf plotting in version 0.15+ that is causing unpredictable and silent errors

araikes commented 2 years ago

Omitting the --output-space and --output-resolution flags and including the --skip-odf-reports flags works but produces a figure-less HTML report.

Interestingly, omitting the --output-space and --output-resolution flags results in an HTML report with figures.

Alex-Clifford commented 2 years ago

I am having an issue with the file result_odf_rois.pklz also not being found. I tried running it with the --skip-odf-reports flag, but it still gets stuck at the same point.

I tried following on from the comment to also omit --output-space and --output-resolution but I was prompted to provide --output-resolution, and it still stalls at the same point with the file not found error. I was wondering if there was another approach to solving this issue?