Problem with eddy_openmp

[k-bingcai]

Hi there,

I was trying out qsiprep v0.14.2 and I encountered this error when it tries to run eddy_openmp

EDDY:::  ECScanManager::GetGlobal2DWIIndexMapping: Global index not dwi
EDDY:::  ECScanClasses.cpp:::  unsigned int EDDY::ECScanManager::GetGlobal2DWIIndexMapping(unsigned int) const:  Exception thrown
EDDY:::  ECScanClasses.h:::  void EDDY::ECScanManager::ApplyDWILocationReference():  Exception thrown
EDDY::: Eddy failed with message EDDY:::  eddy.cpp:::  EDDY::ReplacementManager* EDDY::DoVolumeToVolumeRegistration(const EDDY::EddyCommandLineOptions&, EDDY::E
CScanManager&):  Exception thrown

It is already using the --data_is_shelled flag. I'm not quite sure what is causing this issue. The closest thing that I could find was this thread (https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=FSL;551b44e1.2002) where it says that there may be a programming bug in fsl. Is that the case here?

For some weird reason we can run eddy just fine on our data with fsl 6.0.4 if we call it directly.

Would appreciate any help on this!

Best Regards, Bing Cai

[mattcieslak]

We're using 6.0.3 in qsiprep, so you may need to wait for a new release where we include 6.0.4, unfortunately

[mkleemeyer]

Hey Matt, we are running into the same error here, would you be able to say, when the new release will approximately be? Thanks, Maike

[mattcieslak]

Hi! Sorry this is taking so long. I think there will be a 6.0.4 ready within a month. There are a lot of changes to the infrastructure we're using for development.

In the meantime could you verify that there are

1) no negative values in your DWI series and 2) that your sampling scheme is actually shelled

[octomike]

Btw, there is 6.0.5 out there already, containing another segfault fix in eddy. Any chance we could skip 6.0.4 altogether? We tested a quick and dirty rebuild with:

diff --git a/Dockerfile b/Dockerfile
index c6adf69..0c090ee 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -95,12 +95,12 @@ RUN curl -sSL https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/6.0.1/frees
     --exclude='freesurfer/lib/cuda' \
     --exclude='freesurfer/lib/qt'

-  ENV FSLDIR="/opt/fsl-6.0.3" \
-      PATH="/opt/fsl-6.0.3/bin:$PATH"
+  ENV FSLDIR="/opt/fsl-6.0.5" \
+      PATH="/opt/fsl-6.0.5/bin:$PATH"
   RUN echo "Downloading FSL ..." \
-      && mkdir -p /opt/fsl-6.0.3 \
-      && curl -fsSL --retry 5 https://fsl.fmrib.ox.ac.uk/fsldownloads/fsl-6.0.3-centos6_64.tar.gz \
-      | tar -xz -C /opt/fsl-6.0.3 --strip-components 1 \
+      && mkdir -p /opt/fsl-6.0.5 \
+      && curl -fsSL --retry 5 https://fsl.fmrib.ox.ac.uk/fsldownloads/fsl-6.0.5-centos7_64.tar.gz \
+      | tar -xz -C /opt/fsl-6.0.5 --strip-components 1 \
       --exclude='fsl/doc' \
       --exclude='fsl/data/atlases' \
       --exclude='fsl/data/possum' \
@@ -111,8 +111,8 @@ RUN curl -sSL https://surfer.nmr.mgh.harvard.edu/pub/dist/freesurfer/6.0.1/frees
       && echo "Installing FSL conda environment ..." \
       && sed -i -e "/fsleyes/d" -e "/wxpython/d" \
          ${FSLDIR}/etc/fslconf/fslpython_environment.yml \
-      && bash /opt/fsl-6.0.3/etc/fslconf/fslpython_install.sh -f /opt/fsl-6.0.3 \
-      && find ${FSLDIR}/fslpython/envs/fslpython/lib/python3.7/site-packages/ -type d -name "tests"  -print0 | xargs -0 rm -r \
+      && bash /opt/fsl-6.0.5/etc/fslconf/fslpython_install.sh -f /opt/fsl-6.0.5 \
+      && find ${FSLDIR}/fslpython/envs/fslpython/lib/python3*/site-packages/ -type d -name "tests"  -print0 | xargs -0 rm -r \
       && ${FSLDIR}/fslpython/bin/conda clean --all

 ENV FREESURFER_HOME=/opt/freesurfer \
@@ -225,6 +225,9 @@ RUN mkdir /opt/cmake \

 ENV C3DPATH="/opt/convert3d-nightly" \
     PATH="/opt/convert3d-nightly/bin:$PATH"
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+                    libgnutls30
 RUN echo "Downloading Convert3D ..." \
     && mkdir -p /opt/convert3d-nightly \
     && curl -fsSL --retry 5 https://sourceforge.net/projects/c3d/files/c3d/Nightly/c3d-nightly-Linux-x86_64.tar.gz/download \
@@ -326,7 +329,7 @@ RUN python -c "import amico; amico.core.setup()"
 RUN find $HOME -type d -exec chmod go=u {} + && \
     find $HOME -type f -exec chmod go=u {} +

-RUN ln -s /opt/fsl-6.0.3/bin/eddy_cuda9.1 /opt/fsl-6.0.3/bin/eddy_cuda
+RUN ln -s /opt/fsl-6.0.5/bin/eddy_cuda9.1 /opt/fsl-6.0.5/bin/eddy_cuda

 ENV AFNI_INSTALLDIR=/usr/lib/afni \
     PATH=${PATH}:/usr/lib/afni/bin \

and it seems to build fine. I can make a PR if you want to throw the circle bots at it.

[mattcieslak]

I'm good with skipping 6.0.4! Thanks for your offer on the PR, I'll go ahead and change this in my current branch where a lot of other parts of the circle build have changed

[mattcieslak]

@octomike does the libgnutls30 fix the ssl errors in curl?

[octomike]

@octomike does the libgnutls30 fix the ssl errors in curl?

Yes. It's probably related to that let's encrypt double root issue we saw last year I think. The base container has an older libgnutls30 version and needs an apt-get upgrade or an explicit reinstall

[octomike]

Also note, that for FSL 6.0.5 there is no centos6 "package" anymore and I had to switch to centos7 in the download URL

[mkleemeyer]

Oh no, sorry @mattcieslak, I did not mean to make you feel bad, this was a pure information question to figure out how much time we spend on this. If the next release was next week we would probably just have waited for it;) Our data is not shelled, we just tried with and without that flag to check whether it resolves the issue. However, this was a different thing related to our GPUs with just a very similar looking error but thanks to @octomike is resolved now!

[mattcieslak]

@octomike and @k-bingcai are you using eddy 6.0.5 outside of qsiprep? I'm testing it out and it seems to have some odd multithreading behavior

[octomike]

are you using eddy 6.0.5 outside of qsiprep?

Not personally to give meaningful feedback.

I'm testing it out and it seems to have some odd multithreading behavior

Is it something I can test or validate?

[mattcieslak]

If you're interested, the pennbbl/qsiprep:unstable image has 6.0.5.1 in it!

[arokem]

I am here to report that I was seeing the error mentioned at the beginning of this issue with pennbbl/qsiprep:0.14.3, but it did indeed go away with pennbbl/qsiprep:unstable.

Instead, I am now getting the following error:

Traceback (most recent call last):
--
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
result["result"] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 521, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 639, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
setattr(outputs, key, val)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
value = super(File, self).validate(objekt, name, value, return_pathlike=True)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
self.error(objekt, name, str(value))
File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
raise TraitError(
traits.trait_errors.TraitError: The 'out_parameter' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/cloudknot-user/work/qsiprep_wf/single_subject_FS1_wf/dwi_preproc_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' <class 'str'> was specified.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
outputs = self.aggregate_outputs(runtime)
File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/home/cloudknot-user/work/qsiprep_wf/single_subject_FS1_wf/dwi_preproc_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' for output 'out_parameter' of a ExtendedEddy interface

I don't know if this is a new issue, or related?

[mattcieslak]

@arokem this looks like eddy might be getting killed by the OS before it can write its outputs. Could you try requesting more memory or setting a lower nthreads/omp-nthreads? Either way, this is a poorly reported error

[arokem]

Sorry - was posting in a haste. I think the informative part is the first part where I reported that happily the original issue now seems resolved with the current unstable tag. Will investigate the rest of it.

[mattcieslak]

oops just re-read my post, I was referring to the eddy crash error message as a poorly reported error, not your post!

[arokem]

Ha! All good. I thought you were just being cranky 😄 I did fail to provide some key information such as what exactly I am doing, but let me test things on my end a bit more and tell you what I find out.

[arokem]

I tried increasing RAM to 128GB and set both --nthreads 2 and --omp-nthreads 2.

Attached the full log of this run, in case there are any additional clues in here. FWIW, it's an IVIM scan with some rather odd b-values. I'm going to give this a try with another dataset, with a more conventional acqusition next.

cloudwatch_logs.txt

[arokem]

Thing I learned while making the above comment:

[mattcieslak]

This is definitely crashing on eddy. Are you able to look in the working directories? I wonder if the ivim acquisition doesn't fit cleanly into shells

[mattcieslak]

Closing for now

[arokem]

Sadly, I don't have the working directory accessible anymore, and I have not had a chance to run this again. I'll report back when I have another chance to run this.

[julfou81]

I got a similar issue with qsiprep version 0.16.0RC3 with singularity on HPC:

Command:

singularity run --cleanenv -B /scratch/jsein/BIDS:/work \
 --nv /scratch/jsein/my_images/qsiprep-0.16.0RC3.sif /work/$study  \
 /work/$study/derivatives participant --participant_label $sub     \
 -w /work/temp_data_${study}  --output-resolution 1.2  --fs-license-file /work/freesurfer/license.txt \
  --eddy-config /work/$study/derivatives/eddy_params.json   \
 --b0-threshold 50 --unringing-method mrdegibbs  --denoise-method dwidenoise  \
 --output-space T1w  --distortion-group-merge none 

eddy_params.json file:

{
  "flm": "quadratic",
  "slm": "linear",
  "fep": false,
  "interp": "spline",
  "nvoxhp": 1000,
  "fudge_factor": 10,
  "dont_sep_offs_move": false,
  "dont_peas": false,
  "niter": 5,
  "method": "jac",
  "repol": true,
  "num_threads": 1,
  "is_shelled": true,
  "use_cuda": true,
  "cnr_maps": true,
  "residuals": true,
  "output_type": "NIFTI_GZ",
  "estimate_move_by_susceptibility": true,
  "mporder": 8,
  "slice_order": "/work/TOLD/derivatives/slspec_TOLD.txt",
  "args": "--ol_nstd=5"
}

output log:

subject is 01
base is /scratch/jsein/BIDS/TOLD
study is TOLD
This dataset appears to be BIDS compatible.

        Summary:                Available Tasks:        Available Modalities: 
        70 Files, 4.06GB        TOLD                    T1w                   
        2 - Subjects            VibStim                 T2w                   
        2 - Sessions                                    dwi                   
                                                        bold                  
                                                        sbref                 
                                                        fieldmap              

   If you have any questions, please post on https://neurostars.org/tags/bids.

Making sure the input data is BIDS compliant (warnings can be ignored in most cases).
221004-14:22:22,86 nipype.workflow INFO:
     Running with omp_nthreads=8, nthreads=32
221004-14:22:22,87 nipype.workflow IMPORTANT:

    Running qsiprep version 0.16.0RC3:
      * BIDS dataset path: /work/TOLD.
      * Participant list: ['01'].
      * Run identifier: 20221004-142216_5feddb7f-d0de-4a32-9e87-7489e9bd917b.

221004-14:22:23,379 nipype.utils WARNING:
     A newer version (1.8.4) of nipy/nipype is available. You are using 1.8.1
221004-14:22:24,191 nipype.workflow INFO:
     Combining all dwi files within each available session:
221004-14:22:24,191 nipype.workflow INFO:
        - 0 scans in session pre
221004-14:22:24,191 nipype.workflow INFO:
        - 1 scans in session 01
221004-14:22:24,209 nipype.workflow INFO:
     [{'dwi_series': ['/work/TOLD/sub-01/ses-01/dwi/sub-01_ses-01_dir-AP_dwi.nii.gz'], 'fieldmap_info': {'suffix': 'epi', 'epi': ['/work/TOLD/sub-01/ses-01/fmap/sub-01_ses-01_dir-PA_epi.nii.gz']}, 'dwi_series_pedir': 'j-', 'concatenated_bids_name': 'sub-01_ses-01_dir-AP'}]
221004-14:22:24,292 nipype.workflow IMPORTANT:
     Creating dwi processing workflow "dwi_preproc_ses_01_dir_AP_wf" to produce output sub-01_ses-01_dir-AP (1.17 GB / 119 DWIs). Memory resampled/largemem=1.67/1.86 GB.
221004-14:22:24,294 nipype.workflow INFO:
     Automatically using 5, 5, 5 window for dwidenoise
221004-14:22:24,312 nipype.workflow INFO:
     Using 8 threads in eddy
221004-14:22:27,5 nipype.workflow IMPORTANT:
     Works derived from this qsiprep execution should include the following boilerplate:

Preprocessing was performed using *QSIPrep* 0.16.0RC3,
which is based on *Nipype* 1.8.1
(@nipype1; @nipype2; RRID:SCR_002502).

Anatomical data preprocessing

: A total of 2 T1-weighted (T1w) images were found within the input
BIDS dataset.
All of them were corrected for intensity non-uniformity (INU)
using `N4BiasFieldCorrection` [@n4, ANTs 2.3.1].
A T1w-reference map was computed after registration of
2 T1w images (after INU-correction) using
`mri_robust_template` [FreeSurfer 6.0.1, @fs_template].
The T1w-reference was then skull-stripped using `antsBrainExtraction.sh`
(ANTs 2.3.1), using OASIS as target template.
Spatial normalization to the ICBM 152 Nonlinear Asymmetrical
template version 2009c [@mni, RRID:SCR_008796] was performed
through nonlinear registration with `antsRegistration`
[ANTs 2.3.1, RRID:SCR_004757, @ants], using
brain-extracted versions of both T1w volume and template.
Brain tissue segmentation of cerebrospinal fluid (CSF),
white-matter (WM) and gray-matter (GM) was performed on
the brain-extracted T1w using `FAST` [FSL 6.0.5.1:57b01774, RRID:SCR_002823,
@fsl_fast].

Diffusion data preprocessing

: Any images with a b-value less than 50 s/mm^2 were treated as a *b*=0 image. MP-PCA denoising as implemented in MRtrix3's `dwidenoise`[@dwidenoise1] was applied with a 5-voxel window. After MP-PCA, Gibbs unringing was performed using MRtrix3's `mrdegibbs` [@mrdegibbs]. Following unringing, B1 field inhomogeneity was corrected using `dwibiascorrect` from MRtrix3 with the N4 algorithm [@n4]. After B1 bias correction, the mean intensity of the DWI series was adjusted so all the mean intensity of the b=0 images matched across eachseparate DWI scanning sequence.

FSL (version 6.0.5.1:57b01774)'s eddy was used for head motion correction and Eddy current correction [@anderssoneddy]. Eddy was configured with a $q$-space smoothing factor of 10, a total of 5 iterations, and 1000 voxels used to estimate hyperparameters. A quadratic first level model and a linear second level model were used to characterize Eddy current-related spatial distortion. $q$-space coordinates were forcefully assigned to shells. Field offset was attempted to be separated from subject movement. Shells were aligned post-eddy. Eddy's outlier replacement was run [@eddyrepol]. Data were grouped by slice, only including values from slices determined to contain at least 250 intracerebral voxels. Groups deviating by more than 4 standard deviations from the prediction had their data replaced with imputed values. Slice-to-volume correction was estimated with temporal order 8, 5 iterations, trilinear interpolation and lambda=1.000 [@eddys2v]. Data was collected with reversed phase-encode blips, resulting in pairs of images with distortions going in opposite directions. Here, b=0 reference images with reversed phase encoding directions were used along with an equal number of b=0 images extracted from the DWI scans. From these pairs the susceptibility-induced off-resonance field was estimated using a method similar to that described in [@topup]. The fieldmaps were ultimately incorporated into the Eddy current and head motion correction interpolation. Dynamic susceptibility distortion correction was applied with 10 iterations, lambda=10.00 and spline knot-spacing of 10.00mm [@eddysus]. Final interpolation was performed using the `jac` method.

Several confounding time-series were calculated based on the
preprocessed DWI: framewise displacement (FD) using the
implementation in *Nipype* [following the definitions by @power_fd_dvars].
The head-motion estimates calculated in the correction step were also
placed within the corresponding confounds file. Slicewise cross correlation
was also calculated.
The DWI time-series were resampled to ACPC,
generating a *preprocessed DWI run in ACPC space* with 1.2mm isotropic voxels.

Many internal operations of *QSIPrep* use
*Nilearn* 0.9.1 [@nilearn, RRID:SCR_001362] and
*Dipy* [@dipy].
For more details of the pipeline, see [the section corresponding
to workflows in *QSIPrep*'s documentation](https://qsiprep.readthedocs.io/en/latest/workflows.html "QSIPrep's documentation").

### References

221004-14:22:58,335 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.unwarped_mean" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/unwarped_mean".
221004-14:22:58,348 nipype.workflow INFO:
     [Node] Executing "unwarped_mean" <qsiprep.interfaces.images.IntraModalMerge>
221004-14:23:02,559 nipype.workflow INFO:
     [Node] Finished "unwarped_mean", elapsed time 4.193709s.
221004-14:23:04,128 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.pre_eddy_b0_ref_wf.register_t1_to_raw" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/register_t1_to_raw".
221004-14:23:05,920 nipype.workflow INFO:
     [Node] Executing "register_t1_to_raw" <nipype.interfaces.ants.registration.Registration>
221004-14:24:31,287 nipype.workflow INFO:
     [Node] Finished "register_t1_to_raw", elapsed time 85.168273s.
221004-14:24:32,139 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.pre_eddy_b0_ref_wf.t1_mask_to_b0" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/t1_mask_to_b0".
221004-14:24:33,926 nipype.workflow INFO:
     [Node] Executing "t1_mask_to_b0" <nipype.interfaces.ants.resampling.ApplyTransforms>
221004-14:24:41,206 nipype.workflow INFO:
     [Node] Finished "t1_mask_to_b0", elapsed time 7.08529s.
221004-14:24:43,993 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.pre_eddy_b0_ref_wf.enhance_and_mask_b0" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0".
221004-14:24:44,10 nipype.workflow INFO:
     [Node] Executing "enhance_and_mask_b0" <qsiprep.interfaces.nilearn.EnhanceAndSkullstripB0>
221004-14:24:51,200 nipype.interface WARNING:
     Degenerate Mask case. Using compute_epi_mask
221004-14:24:53,998 nipype.interface WARNING:
     Masking appears to have failed. Using a backup method
221004-14:24:58,77 nipype.workflow INFO:
     [Node] Finished "enhance_and_mask_b0", elapsed time 14.050999s.
221004-14:25:02,166 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.b0_ref_to_lps" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/b0_ref_to_lps".
221004-14:25:02,166 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.b0_ref_mask_to_lps" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/b0_ref_mask_to_lps".
221004-14:25:02,229 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.b0_ref_brain_to_lps" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/b0_ref_brain_to_lps".
221004-14:25:02,236 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.eddy" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy".
221004-14:25:02,279 nipype.workflow INFO:
     [Node] Executing "b0_ref_to_lps" <qsiprep.interfaces.images.ConformDwi>
221004-14:25:02,294 nipype.workflow INFO:
     [Node] Executing "b0_ref_mask_to_lps" <qsiprep.interfaces.images.ConformDwi>
221004-14:25:02,302 nipype.workflow INFO:
     [Node] Executing "eddy" <qsiprep.interfaces.eddy.ExtendedEddy>
221004-14:25:02,337 nipype.workflow INFO:
     [Node] Executing "b0_ref_brain_to_lps" <qsiprep.interfaces.images.ConformDwi>
221004-14:25:02,354 nipype.interface INFO:
     Re-orienting /work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_mask.nii.gz to LPS
221004-14:25:02,375 nipype.workflow INFO:
     [Node] Finished "b0_ref_mask_to_lps", elapsed time 0.06279s.
221004-14:25:02,672 nipype.interface INFO:
     Re-orienting /work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_unbiasedsharpened.nii.gz to LPS
221004-14:25:02,743 nipype.interface INFO:
     Re-orienting /work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_brain.nii.gz to LPS
221004-14:25:02,946 nipype.workflow INFO:
     [Node] Finished "b0_ref_to_lps", elapsed time 0.649495s.
221004-14:25:03,46 nipype.workflow INFO:
     [Node] Finished "b0_ref_brain_to_lps", elapsed time 0.690127s.
221004-14:25:06,58 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.b0_anat_coreg.b0_to_anat" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/b0_anat_coreg/b0_to_anat".
221004-14:25:06,59 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.b0_anat_coreg.b0_to_anat".
221004-14:25:06,76 nipype.workflow INFO:
     [Node] Executing "b0_to_anat" <qsiprep.interfaces.niworkflows.ANTSRegistrationRPT>
221004-14:27:16,51 nipype.workflow INFO:
     [Node] Finished "eddy", elapsed time 133.728166s.
221004-14:27:16,51 nipype.workflow WARNING:
     Storing result file without outputs
221004-14:27:16,53 nipype.workflow WARNING:
     [Node] Error on "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.eddy" (/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy)
221004-14:27:16,217 nipype.workflow ERROR:
     Node eddy failed to run on host gpu015.cluster.
221004-14:27:16,222 nipype.workflow ERROR:
     Saving crash info to /work/TOLD/derivatives/qsiprep/sub-01/log/20221004-142216_5feddb7f-d0de-4a32-9e87-7489e9bd917b/crash-20221004-142716-jsein-eddy-21ce91c6-bc21-41c2-8834-f1b885094e73.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_parameter' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' for output 'out_parameter' of a ExtendedEddy interface

221004-14:28:53,823 nipype.interface INFO:
     Report - setting fixed (/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/anat_preproc_wf/skullstrip_wf/rigid_acpc_resample_brain/highres001_BrainExtractionBrain_trans.nii.gz) and moving (/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/b0_anat_coreg/b0_to_anat/transform_Warped.nii.gz) images
221004-14:29:06,188 nipype.workflow INFO:
     [Node] Finished "b0_to_anat", elapsed time 239.901573s.
221004-14:29:08,123 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.map_seg" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/fmap_unwarp_report_wf/map_seg".
221004-14:29:08,124 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.map_seg".
221004-14:29:08,138 nipype.workflow INFO:
     [Node] Executing "map_seg" <qsiprep.niworkflows.interfaces.fixes.FixHeaderApplyTransforms>
221004-14:29:16,158 nipype.workflow INFO:
     [Node] Finished "map_seg", elapsed time 7.858042s.
221004-14:29:16,497 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.sel_wm" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/fmap_unwarp_report_wf/sel_wm".
221004-14:29:16,498 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.sel_wm".
221004-14:29:16,511 nipype.workflow INFO:
     [Node] Executing "sel_wm" <nipype.interfaces.utility.wrappers.Function>
221004-14:29:16,626 nipype.workflow INFO:
     [Node] Finished "sel_wm", elapsed time 0.09857s.
221004-14:29:19,969 nipype.workflow INFO:
     [Node] Setting-up "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.dwi_rpt" in "/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/fmap_unwarp_report_wf/dwi_rpt".
221004-14:29:19,970 nipype.workflow INFO:
     [Node] Outdated cache found for "qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.fmap_unwarp_report_wf.dwi_rpt".
221004-14:29:19,982 nipype.workflow INFO:
     [Node] Executing "dwi_rpt" <qsiprep.niworkflows.interfaces.registration.SimpleBeforeAfterRPT>
221004-14:29:33,201 nipype.workflow INFO:
     [Node] Finished "dwi_rpt", elapsed time 13.201134s.
221004-14:29:36,413 nipype.workflow ERROR:
     could not run node: qsiprep_wf.single_subject_01_wf.dwi_preproc_ses_01_dir_AP_wf.hmc_sdc_wf.eddy

error log:

[WARNING] This document format requires a nonempty <title> element.
  Please specify either 'title' or 'pagetitle' in the metadata.
  Falling back to 'CITATION'
 file none does not exist . 
 file none does not exist . 
QSIPrep failed: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_parameter' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' for output 'out_parameter' of a ExtendedEddy interface

Traceback (most recent call last):
  File "/usr/local/miniconda/bin/qsiprep", line 8, in <module>
    sys.exit(main())
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/run.py", line 659, in main
    qsiprep_wf.run(**plugin_settings)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/base.py", line 212, in run
    raise error from cause
RuntimeError: Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy.

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/usr/local/miniconda/lib/python3.8/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_parameter' trait of an ExtendedEddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/work/temp_data_TOLD/qsiprep_wf/single_subject_01_wf/dwi_preproc_ses_01_dir_AP_wf/hmc_sdc_wf/eddy/eddy_corrected.eddy_parameters' for output 'out_parameter' of a ExtendedEddy interface

Sentry is attempting to send 4 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit

[julfou81]

This is very strange. It is a dataset for which we acquired the set of directions in AP and PA. QSIPREP worked file for the full data set. For the sake of tests, I am trying to compare it with the dataset in AP only (corrected for topup with the b0 PA), and the dataset in PA only (corrected for topup with the b0 AP). QSIPREP worked well with the PA set but failed systematically for the AP set.

Since as you mentioned the bug is poorly reported, I tried to launch eddy_openmp on my personal laptop (FSL 6.0.5) on the same data and here is the output:

eddy --ol_nstd=5 --cnr_maps --estimate_move_by_susceptibility --field=fieldmap_HZ --field_mat=topup_reg_image_flirt.mat --flm=quadratic --ff=10.0 --acqp=eddy_acqp.txt --bvals=sub-01_ses-01_dir-AP_dwi.bval --bvecs=sub-01_ses-01_dir-AP_dwi.bvec --imain=sub-01_ses-01_dir-AP_dwi_denoised_mrdegibbs_N4.nii.gz --index=eddy_index.txt --mask=topup_imain_corrected_avg_mask.nii.gz --interp=spline --data_is_shelled --resamp=jac --mporder=8 --niter=5 --nvoxhp=1000 --out=eddy_corrected --repol --residuals --slm=linear -v
Reading images
Performing volume-to-volume registration
Running Register
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 0, Total mss = 30.8269
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 1, Total mss = 20.4061
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 2, Total mss = 18.5778
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 3, Total mss = 17.5641
Loading prediction maker
Evaluating prediction maker model
Calculating parameter updates
Iter: 4, Total mss = 17.0193
Setting scan 40 as b0 shape-reference.
Running sm.ApplyB0LocationReference
Running sm.PolateB0MovPar
Running Register
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 1.740062 
0.371253 
0.545023 
8.038973 
3.867351 
3.089012 

Calculating parameter updates
exitIter: 0, Total mss = 3.931948
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 1.591906 
0.408377 
0.469477 
14.133289 
3.604093 
2.597913 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.323975 
0.302783 
0.466343 
6.255050 
4.258123 
3.307055 

Calculating parameter updates
Iter: 1, Total mss = 3.021127
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.269008 
0.290803 
0.495311 
5.987899 
4.283569 
3.266975 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.262518 
0.276843 
0.489414 
5.826456 
4.381449 
3.256601 

Calculating parameter updates
Iter: 2, Total mss = 2.917337
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.432599 
0.283886 
0.496786 
6.313681 
4.423093 
3.494168 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.211105 
0.278536 
0.479018 
6.126618 
4.219609 
3.204969 

Calculating parameter updates
Iter: 3, Total mss = 2.828135
Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.480159 
0.257420 
0.466856 
5.911317 
4.372031 
3.341397 

Loading prediction maker
Evaluating prediction maker model
Estimated hyperparameters: 2.263637 
0.302829 
0.459527 
5.946027 
4.233073 
3.187096 

Calculating parameter updates
Iter: 4, Total mss = 2.720007

EDDY::: Eddy failed with message EDDY:::  eddy.cpp:::  EDDY::ReplacementManager *EDDY::DoVolumeToVolumeRegistration(const EDDY::EddyCommandLineOptions &, EDDY::ECScanManager &):  Exception thrown

[mattcieslak]

You should open a new issue for this. The original issue here was that eddy was crashing and not letting nipype know about it, so an error is raised when looking for one of its output files

[julfou81]

ok, thanks for your quick response, will do!