EDDY_CUDA Failure: GPU can only use 1 CPU thread

[amirhusseinab]

Summary

Running with an eddy_config file to utilize CUDA and mporder results in an error in eddy_cuda step:

EddyInputError: The version compiled for GPU can only use 1 CPU thread (i.e. --nthr=1)
Terminating program

Additional details

• QSIPrep version: 0.22.0
• Docker version:
• Singularity version:

What were you trying to do?

The command I used:

singularity run --nv --cleanenv -B $TMPDIR:$TMPDIR \
-B /path/to/input/bids/directory:/input \
-B /path/to/toolboxes/folder:/toolboxes \
toolboxes/qsiprep-v0.22.0.sif /input/ /input/derivatives/qsiprep_0.22.0/ participant \
--fs-license-file toolboxes/license.txt \
--participant-label $subjid --output-resolution 1.25 \
--denoise-method dwidenoise --unringing-method mrdegibbs \
--hmc-model eddy --eddy-config /input/code/eddy_params.json --pepolar-method TOPUP \
--anat-modality T1w --anatomical-template MNI152NLin2009cAsym --b0_to_t1w_transform Rigid \
--freesurfer-input /input/derivatives/freesurfer -w /path/to/work \
--stop-on-first-crash --nthreads 30 --skip_bids_validation

The config file:

{
  "flm": "quadratic",
  "slm": "linear",
  "fep": false,
  "interp": "spline",
  "nvoxhp": 1000,
  "fudge_factor": 10,
  "dont_sep_offs_move": false,
  "dont_peas": false,
  "niter": 5,
  "method": "jac",
  "repol": true,
  "num_threads": 1,
  "is_shelled": true,
  "use_cuda": true,
  "cnr_maps": true,
  "residuals": true,
  "output_type": "NIFTI_GZ",
  "estimate_move_by_susceptibility": true,
  "mporder": 6,
  "args": "--ol_nstd=5"
}

What did you expect to happen?

To have a successful preprocessing :)

What actually happened?

Despite explicitly setting num_threads to 1 in the eddy_config, It prints out:

240725-14:26:19,338 nipype.workflow INFO:
     Using 8 threads in eddy

And later it fails with an error:

EddyInputError: The version compiled for GPU can only use 1 CPU thread (i.e. --nthr=1)
Terminating program

In the output log file, the --nthr flag in eddy_cuda is also set to 8.

Reproducing the bug

[amirhusseinab]

Update:

Setting --omp-nthreads 1 fixes the issue as expected, but the processing time of non-eddy functions is a lot.

[araikes]

This was fixed in #788 (merged yesterday)

[amirhusseinab]

I had this issue with 0.22.0 - downloaded today.

[amirhusseinab]

I think this is also relevant - on overriding the default eddy with eddy_config.json, in the boilerplate it says: "Groups deviating by more than 4 standard deviations from the prediction had their data replaced with imputed values" I think it is set by ol_nstd argument in eddy_config which I have set to 5 rather than the default 4.