error with eddy (open_mp or cuda)

julfou81 commented 2 years ago

This an issue for a dataset for which we acquired the same set set of directions in AP and PA. QSIPREP worked fine for the full data set. For the sake of tests, I am trying to compare it with the dataset in AP only (corrected for topup with the b0 PA), and the dataset in PA only (corrected for topup with the b0 AP). QSIPREP worked well with the PA set but failed systematically for the AP set.

I am using qsiprep version 0.16.0RC3 with singularity on HPC using slurs: The exact same qsiprep and slurs settings setting was working well for several subjects and the other set of data of the same subject as I was explaining above

Command:

singularity run --cleanenv -B /scratch/jsein/BIDS:/work \
 --nv /scratch/jsein/my_images/qsiprep-0.16.0RC3.sif /work/$study  \
 /work/$study/derivatives participant --participant_label $sub     \
 -w /work/temp_data_${study}  --output-resolution 1.2  --fs-license-file /work/freesurfer/license.txt \
  --eddy-config /work/$study/derivatives/eddy_params.json   \
 --b0-threshold 50 --unringing-method mrdegibbs  --denoise-method dwidenoise  \
 --output-space T1w  --distortion-group-merge none

eddy_params.json file:

{
  "flm": "quadratic",
  "slm": "linear",
  "fep": false,
  "interp": "spline",
  "nvoxhp": 1000,
  "fudge_factor": 10,
  "dont_sep_offs_move": false,
  "dont_peas": false,
  "niter": 5,
  "method": "jac",
  "repol": true,
  "num_threads": 1,
  "is_shelled": true,
  "use_cuda": true,
  "cnr_maps": true,
  "residuals": true,
  "output_type": "NIFTI_GZ",
  "estimate_move_by_susceptibility": true,
  "mporder": 8,
  "slice_order": "/work/TOLD/derivatives/slspec_TOLD.txt",
  "args": "--ol_nstd=5"
}

output log:

subject is 01
base is /scratch/jsein/BIDS/TOLD
study is TOLD
[32mThis dataset appears to be BIDS compatible.[39m

        [34m[4mSummary:[24m[39m                [34m[4mAvailable Tasks:[24m[39m        [34m[4mAvailable Modalities:[24m[39m 
        70 Files, 4.06GB        TOLD                    T1w                   
        2 - Subjects            VibStim                 T2w                   
        2 - Sessions                                    dwi                   
                                                        bold                  
                                                        sbref                 
                                                        fieldmap              

[36m   If you have any questions, please post on https://neurostars.org/tags/bids.[39m

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
221004-14:22:22,86 nipype.workflow INFO:
     Running with omp_nthreads=8, nthreads=32
221004-14:22:22,87 nipype.workflow IMPORTANT:

    Running qsiprep version 0.16.0RC3:
      * BIDS dataset path: /work/TOLD.
      * Participant list: ['01'].
      * Run identifier: 20221004-142216_5feddb7f-d0de-4a32-9e87-7489e9bd917b.

221004-14:22:23,379 nipype.utils WARNING:
     A newer version (1.8.4) of nipy/nipype is available. You are using 1.8.1
221004-14:22:24,191 nipype.workflow INFO:
     Combining all dwi files within each available session:
221004-14:22:24,191 nipype.workflow INFO:
        - 0 scans in session pre
221004-14:22:24,191 nipype.workflow INFO:
        - 1 scans in session 01
221004-14:22:24,209 nipype.workflow INFO:
     [{'dwi_series': ['/work/TOLD/sub-01/ses-01/dwi/sub-01_ses-01_dir-AP_dwi.nii.gz'], 'fieldmap_info': {'suffix': 'epi', 'epi': ['/work/TOLD/sub-01/ses-01/fmap/sub-01_ses-01_dir-PA_epi.nii.gz']}, 'dwi_series_pedir': 'j-', 'concatenated_bids_name': 'sub-01_ses-01_dir-AP'}]
221004-14:22:24,292 nipype.workflow IMPORTANT:
     Creating dwi processing workflow "dwi_preproc_ses_01_dir_AP_wf" to produce output sub-01_ses-01_dir-AP (1.17 GB / 119 DWIs). Memory resampled/largemem=1.67/1.86 GB.
221004-14:22:24,294 nipype.workflow INFO:
     Automatically using 5, 5, 5 window for dwidenoise
221004-14:22:24,312 nipype.workflow INFO:
     Using 8 threads in eddy
221004-14:22:27,5 nipype.workflow IMPORTANT:
     Works derived from this qsiprep execution should include the following boilerplate:

Preprocessing was performed using *QSIPrep* 0.16.0RC3,
which is based on *Nipype* 1.8.1
(@nipype1; @nipype2; RRID:SCR_002502).

Anatomical data preprocessing

: A total of 2 T1-weighted (T1w) images were found within the input
BIDS dataset.
All of them were corrected for intensity non-uniformity (INU)
using `N4BiasFieldCorrection` [@n4, ANTs 2.3.1].
A T1w-reference map was computed after registration of
2 T1w images (after INU-correction) using
`mri_robust_template` [FreeSurfer 6.0.1, @fs_template].
The T1w-reference was then skull-stripped using `antsBrainExtraction.sh`
(ANTs 2.3.1), using OASIS as target template.
Spatial normalization to the ICBM 152 Nonlinear Asymmetrical
template version 2009c [@mni, RRID:SCR_008796] was performed
through nonlinear registration with `antsRegistration`
[ANTs 2.3.1, RRID:SCR_004757, @ants], using
brain-extracted versions of both T1w volume and template.
Brain tissue segmentation of cerebrospinal fluid (CSF),
white-matter (WM) and gray-matter (GM) was performed on
the brain-extracted T1w using `FAST` [FSL 6.0.5.1:57b01774, RRID:SCR_002823,
@fsl_fast].

Diffusion data preprocessing

: Any images with a b-value less than 50 s/mm^2 were treated as a *b*=0 image. MP-PCA denoising as implemented in MRtrix3's `dwidenoise`[@dwidenoise1] was applied with a 5-voxel window. After MP-PCA, Gibbs unringing was performed using MRtrix3's `mrdegibbs` [@mrdegibbs]. Following unringing, B1 field inhomogeneity was corrected using `dwibiascorrect` from MRtrix3 with the N4 algorithm [@n4]. After B1 bias correction, the mean intensity of the DWI series was adjusted so all the mean intensity of the b=0 images matched across eachseparate DWI scanning sequence.

FSL (version 6.0.5.1:57b01774)'s eddy was used for head motion correction and Eddy current correction [@anderssoneddy]. Eddy was configured with a $q$-space smoothing factor of 10, a total of 5 iterations, and 1000 voxels used to estimate hyperparameters. A quadratic first level model and a linear second level model were used to characterize Eddy current-related spatial distortion. $q$-space coordinates were forcefully assigned to shells. Field offset was attempted to be separated from subject movement. Shells were aligned post-eddy. Eddy's outlier replacement was run [@eddyrepol]. Data were grouped by slice, only including values from slices determined to contain at least 250 intracerebral voxels. Groups deviating by more than 4 standard deviations from the prediction had their data replaced with imputed values. Slice-to-volume correction was estimated with temporal order 8, 5 iterations, trilinear interpolation and lambda=1.000 [@eddys2v]. Data was collected with reversed phase-encode blips, resulting in pairs of images with distortions going in opposite directions. Here, b=0 reference images with reversed phase encoding directions were used along with an equal number of b=0 images extracted from the DWI scans. From these pairs the susceptibility-induced off-resonance field was estimated using a method similar to that described in [@topup]. The fieldmaps were ultimately incorporated into the Eddy current and head motion correction interpolation. Dynamic susceptibility distortion correction was applied with 10 iterations, lambda=10.00 and spline knot-spacing of 10.00mm [@eddysus]. Final interpolation was performed using the `jac` method.

Several confounding time-series were calculated based on the
preprocessed DWI: framewise displacement (FD) using the
implementation in *Nipype* [following the definitions by @power_fd_dvars].
The head-motion estimates calculated in the correction step were also
placed within the corresponding confounds file. Slicewise cross correlation
was also calculated.
The DWI time-series were resampled to ACPC,
generating a *preprocessed DWI run in ACPC space* with 1.2mm isotropic voxels.

Many internal operations of *QSIPrep* use
*Nilearn* 0.9.1 [@nilearn, RRID:SCR_001362] and
*Dipy* [@dipy].
For more details of the pipeline, see [the section corresponding
to workflows in *QSIPrep*'s documentation](https://qsiprep.readthedocs.io/en/latest/workflows.html "QSIPrep's documentation").

### References

221004-14:22:58,335 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.unwarped_mean" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/unwarped_mean".
221004-14:22:58,348 nipype.workflow INFO:
     [Node] Executing "unwarped_mean" <qsiprep.interfaces.images.IntraModalMerge>
221004-14:23:02,559 nipype.workflow INFO:
     [Node] Finished "unwarped_mean", elapsed time 4.193709s.
221004-14:23:04,128 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.pre_eddy_b0_ref_wf.register_t1_to_raw" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/register_t1_to_raw".
221004-14:23:05,920 nipype.workflow INFO:
     [Node] Executing "register_t1_to_raw" <nipype.interfaces.ants.registration.Registration>
221004-14:24:31,287 nipype.workflow INFO:
     [Node] Finished "register_t1_to_raw", elapsed time 85.168273s.
221004-14:24:32,139 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.pre_eddy_b0_ref_wf.t1_mask_to_b0" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/t1_mask_to_b0".
221004-14:24:33,926 nipype.workflow INFO:
     [Node] Executing "t1_mask_to_b0" <nipype.interfaces.ants.resampling.ApplyTransforms>
221004-14:24:41,206 nipype.workflow INFO:
     [Node] Finished "t1_mask_to_b0", elapsed time 7.08529s.
221004-14:24:43,993 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.pre_eddy_b0_ref_wf.enhance_and_mask_b0" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0".
221004-14:24:44,10 nipype.workflow INFO:
     [Node] Executing "enhance_and_mask_b0" <qsiprep.interfaces.nilearn.EnhanceAndSkullstripB0>
221004-14:24:51,200 nipype.interface WARNING:
     Degenerate Mask case. Using compute_epi_mask
221004-14:24:53,998 nipype.interface WARNING:
     Masking appears to have failed. Using a backup method
221004-14:24:58,77 nipype.workflow INFO:
     [Node] Finished "enhance_and_mask_b0", elapsed time 14.050999s.
221004-14:25:02,166 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.b0_ref_to_lps" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/b0_ref_to_lps".
221004-14:25:02,166 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.b0_ref_mask_to_lps" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/b0_ref_mask_to_lps".
221004-14:25:02,229 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.b0_ref_brain_to_lps" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/b0_ref_brain_to_lps".
221004-14:25:02,236 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.eddy" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy".
221004-14:25:02,279 nipype.workflow INFO:
     [Node] Executing "b0_ref_to_lps" <qsiprep.interfaces.images.ConformDwi>
221004-14:25:02,294 nipype.workflow INFO:
     [Node] Executing "b0_ref_mask_to_lps" <qsiprep.interfaces.images.ConformDwi>
221004-14:25:02,302 nipype.workflow INFO:
     [Node] Executing "eddy" <qsiprep.interfaces.eddy.ExtendedEddy>
221004-14:25:02,337 nipype.workflow INFO:
     [Node] Executing "b0_ref_brain_to_lps" <qsiprep.interfaces.images.ConformDwi>
221004-14:25:02,354 nipype.interface INFO:
     Re-orienting /work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_mask.nii.gz to LPS
221004-14:25:02,375 nipype.workflow INFO:
     [Node] Finished "b0_ref_mask_to_lps", elapsed time 0.06279s.
221004-14:25:02,672 nipype.interface INFO:
     Re-orienting /work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_unbiasedsharpened.nii.gz to LPS
221004-14:25:02,743 nipype.interface INFO:
     Re-orienting /work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_brain.nii.gz to LPS
221004-14:25:02,946 nipype.workflow INFO:
     [Node] Finished "b0_ref_to_lps", elapsed time 0.649495s.
221004-14:25:03,46 nipype.workflow INFO:
     [Node] Finished "b0_ref_brain_to_lps", elapsed time 0.690127s.
221004-14:25:06,58 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.b0_anat_coreg.b0_to_anat" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/b0_anat_coreg/b0_to_anat".
221004-14:25:06,59 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.b0_anat_coreg.b0_to_anat".
221004-14:25:06,76 nipype.workflow INFO:
     [Node] Executing "b0_to_anat" <qsiprep.interfaces.niworkflows.ANTSRegistrationRPT>
221004-14:27:16,51 nipype.workflow INFO:
     [Node] Finished "eddy", elapsed time 133.728166s.
221004-14:27:16,51 nipype.workflow WARNING:
     Storing result file without outputs
221004-14:27:16,53 nipype.workflow WARNING:
     [Node] Error on "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.eddy" (/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy)
221004-14:27:16,217 nipype.workflow ERROR:
     Node eddy failed to run on host gpu015.cluster.
221004-14:27:16,222 nipype.workflow ERROR:
     Saving crash info to /work/TOLD/derivatives/qsiprep/sub-01/log/20221004-142216_5feddb7f-d0de-4a32-9e87-7489e9bd917b/crash-20221004-142716-jsein-eddy-21ce91c6-bc21-41c2-8834-f1b885094e73.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_parameter' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' for output 'out_parameter' of a ExtendedEddy interface

221004-14:28:53,823 nipype.interface INFO:
     Report - setting fixed (/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/skullstrip_wf/rigid_acpc_resample_brain/highres001_BrainExtractionBrain_trans.nii.gz) and moving (/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/b0_anat_coreg/b0_to_anat/transform_Warped.nii.gz) images
221004-14:29:06,188 nipype.workflow INFO:
     [Node] Finished "b0_to_anat", elapsed time 239.901573s.
221004-14:29:08,123 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.map_seg" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/fmap_unwarp_report_wf/map_seg".
221004-14:29:08,124 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.map_seg".
221004-14:29:08,138 nipype.workflow INFO:
     [Node] Executing "map_seg" <qsiprep.niworkflows.interfaces.fixes.FixHeaderApplyTransforms>
221004-14:29:16,158 nipype.workflow INFO:
     [Node] Finished "map_seg", elapsed time 7.858042s.
221004-14:29:16,497 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.sel_wm" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/fmap_unwarp_report_wf/sel_wm".
221004-14:29:16,498 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.sel_wm".
221004-14:29:16,511 nipype.workflow INFO:
     [Node] Executing "sel_wm" <nipype.interfaces.utility.wrappers.Function>
221004-14:29:16,626 nipype.workflow INFO:
     [Node] Finished "sel_wm", elapsed time 0.09857s.
221004-14:29:19,969 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.dwi_rpt" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/fmap_unwarp_report_wf/dwi_rpt".
221004-14:29:19,970 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.dwi_rpt".
221004-14:29:19,982 nipype.workflow INFO:
     [Node] Executing "dwi_rpt" <qsiprep.niworkflows.interfaces.registration.SimpleBeforeAfterRPT>
221004-14:29:33,201 nipype.workflow INFO:
     [Node] Finished "dwi_rpt", elapsed time 13.201134s.
221004-14:29:36,413 nipype.workflow ERROR:
     could not run node: qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.eddy

error log:

[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'CITATION'
 file none does not exist . 
 file none does not exist . 
QSIPrep failed: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_parameter' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' for output 'out_parameter' of a ExtendedEddy interface

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/qsiprep", line 8, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/run.py", line 659, in main
    qsiprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/base.py", line 212, in run
    raise error from cause
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_parameter' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' for output 'out_parameter' of a ExtendedEddy interface

Sentry is attempting to send 4 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

To better understand why eddy is crashing, I tried to launch eddy_openmp on my personal laptop (FSL 6.0.5) on the same data prepared by qsiprep (exact same files) and here is the output:

eddy --ol_nstd=5 --cnr_maps --estimate_move_by_susceptibility --field=fieldmap_HZ --field_mat=topup_reg_image_flirt.mat --flm=quadratic --ff=10.0 --acqp=eddy_acqp.txt --bvals=sub-01_ses-01_dir-AP_dwi.bval --bvecs=sub-01_ses-01_dir-AP_dwi.bvec --imain=sub-01_ses-01_dir-AP_dwi_denoised_mrdegibbs_N4.nii.gz --index=eddy_index.txt --mask=topup_imain_corrected_avg_mask.nii.gz --interp=spline --data_is_shelled --resamp=jac --mporder=8 --niter=5 --nvoxhp=1000 --out=eddy_corrected --repol --residuals --slm=linear -v
Reading images
Performing volume-to-volume registration
Running Register
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 30.8269
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 20.4061
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 18.5778
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 17.5641
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 17.0193
Setting scan 40 as b0 shape-reference.
Running sm.ApplyB0LocationReference
Running sm.PolateB0MovPar
Running Register
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 1.740062 
0.371253 
0.545023 
8.038973 
3.867351 
3.089012 

Calculating parameter updates
exitIter: 0, Total mss = 3.931948
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 1.591906 
0.408377 
0.469477 
14.133289 
3.604093 
2.597913 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.323975 
0.302783 
0.466343 
6.255050 
4.258123 
3.307055 

Calculating parameter updates
Iter: 1, Total mss = 3.021127
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.269008 
0.290803 
0.495311 
5.987899 
4.283569 
3.266975 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.262518 
0.276843 
0.489414 
5.826456 
4.381449 
3.256601 

Calculating parameter updates
Iter: 2, Total mss = 2.917337
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.432599 
0.283886 
0.496786 
6.313681 
4.423093 
3.494168 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.211105 
0.278536 
0.479018 
6.126618 
4.219609 
3.204969 

Calculating parameter updates
Iter: 3, Total mss = 2.828135
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.480159 
0.257420 
0.466856 
5.911317 
4.372031 
3.341397 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.263637 
0.302829 
0.459527 
5.946027 
4.233073 
3.187096 

Calculating parameter updates
Iter: 4, Total mss = 2.720007

EDDY::: Eddy failed with message EDDY:::  eddy.cpp:::  EDDY::ReplacementManager *EDDY::DoVolumeToVolumeRegistration(const EDDY::EddyCommandLineOptions &, EDDY::ECScanManager &):  Exception thrown

mattcieslak commented 2 years ago

Do both sequences have enough b>0 images for eddy to run?

Also, you can keep both AP and PA scans in the dwi/ directory and use the --separate-all-dwis to have them processed on their own.

julfou81 commented 2 years ago

Thank you for the tip!

Yes both sets in AP and PA consist of 109 volumes total, of which: 7 b=0, 6b=300, 32 b=1000 and 64 b=2000.

For running each subset separately, I rebuilt a BIDS tree by putting the only b=0 in opposite direction in a fmap/ folder, renaming them with _epi suffix and filling the json file with IntendedFor field. This was working well for the PA set (with the b0 AP put in the fmap/ folder), but eddy crashes for the second set (full AP in dwi/ folder and PA b0 in fmap/ folder).

When you use --separate-all-dwis, does topup uses the b0 from the other set to run?

mattcieslak commented 2 years ago

yes, it will look in the dwi files to see if any have b=0 images that are suitable for distortion correction. Don't bother putting anything in the fmap/ directory

julfou81 commented 2 years ago

I noticed something interesting when trying to use --distortion-group-merge and --separate-all-dwis for my dataset with the same set of directions repeated for both encoding direction (AP and PA):

for my first try, I saw in the working directory that the AP and PA sets were indeed processed separately, but it seems that topup is not run and the execution crashes during eddy execution with the same error as described above: FileNotFoundError: No such file or directory '/work/temp_data_DWImergenone_PhantomPainBrain/qsiprep_wf/single_subject_CTS02_wf/dwi_preproc_ses_BRpre_dir_PA_wf/hmc_sdc_wf/eddy/eddy_corrected.nii.gz' for output 'out_corrected' of a ExtendedEddy interface
using the exact same dwi folder, I added an fmap/ folder where I extracted the b0s for each dwi set (7 for each PE direction), renamed them with _epi suffix and use ÌntendedForfields to specify for which data each b0 is attributed (b0_APfor _dir-PA_dwi andb0_PAfordir-AP_dwi`). In that case, qsiprep is completed with no error, topup is run but the resulting preprocessed dwi images is a concatenation of both AP and PA dwi volumes.

mattcieslak commented 10 months ago

This should be fixed in recent releases

PennLINC / qsiprep

error with eddy (open_mp or cuda) #472