search

aramis-lab / clinica

Software platform for clinical neuroimaging studies
http://www.clinica.run/
Other
221 stars 74 forks source link

Cannot conduct "dwi-preprocessing-using-t1" without using cuda #1075

Closed felicityshaw closed 2 months ago

felicityshaw commented 8 months ago

Hello,

Thanks for your development and sharing of Clinica; it's a very useful tool.

Recently, while attempting the "dwi-preprocessing-using-t1" pipeline without utilizing CUDA, I encountered an unexpected error related to the inability to locate the eddy_corrected file. In an effort to resolve this issue, I referred to solutions for similar issues posted on this git. Following the guidance, I executed the command detailed in the "command.txt" file within the corresponding folder, which initiates with "eddy". Unfortunately, this resulted in a "ValueError: could not convert string to float: ''". Additionally, executing the simple "eddy" command in the terminal was unsuccessful until I substituted "eddy" with "eddy_cpu".

Could you please advise on how to address this issue? Thank you in advance for your assistance.

NicolasGensollen commented 8 months ago

Hi @felicityshaw

Thanks for reporting. Could you provide a bit more information about your setup and what you were trying to do ? (OS name and version, Python version, Clinica version, what the input data looks like, the command you ran, the error you got when running the pipeline, and the error you got running the command in the command.txt file)

felicityshaw commented 8 months ago

Sure.

OS: Ubuntu 20.04.6

Python: 3.11.6

Input data: DWI in BIDS format

The command I ran: clinica run dwi-preprocessing-using-t1 --n_procs 8 -wd wd_copy BIDS_copy CAPS_copy

Error I got:

RuntimeError: Traceback (most recent call last):
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy_fsl.

Traceback (most recent call last):
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 454, in aggregate_outputs
    setattr(outputs, key, val)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
    value = super(File, self).validate(objekt, name, value, return_pathlike=True)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
    self.error(objekt, name, str(value))
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
    raise TraitError(
traits.trait_errors.TraitError: The 'out_corrected' trait of an EddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/local/PARTNERS/mount_file/wd_copy/dwi-preprocessing-using-t1/eddy_fsl/854443cb478db7cdc37ae2446543b7934a6a857d/eddy_fsl/eddy_corrected.nii.gz' <class 'str'> was specified.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 401, in run
    outputs = self.aggregate_outputs(runtime)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 461, in aggregate_outputs
    raise FileNotFoundError(msg)
FileNotFoundError: No such file or directory '/home/local/PARTNERS/mount_file/wd_copy/dwi-preprocessing-using-t1/eddy_fsl/854443cb478db7cdc37ae2446543b7934a6a857d/eddy_fsl/eddy_corrected.nii.gz' for output 'out_corrected' of a Eddy interface

Go to folder: 

/home/local/PARTNERS/mount_file/wd_copy/dwi-preprocessing-using-t1/eddy_fsl/854443cb478db7cdc37ae2446543b7934a6a857d/eddy_fsl/

Files in this folder: image

Content in 'command.txt':

eddy --flm=quadratic --ff=10.0 --acqp=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/acq.txt --bvals=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/sub-ADNI003S4119_ses-M006_dwi_merged.bval --bvecs=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/sub-ADNI003S4119_ses-M006_dwi_merged.bvec --imain=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/sub-ADNI003S4119_ses-M006_dwi_merged.nii.gz --index=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/index.txt --mask=/home/local/PARTNERS/mount_file/wd_copy/dwi-preprocessing-using-t1/eddy_fsl/854443cb478db7cdc37ae2446543b7934a6a857d/mask_reference_b0/reference_b0_volume_brain_mask.nii.gz --interp=spline --resamp=jac --niter=5 --nvoxhp=1000 --out=/home/local/PARTNERS/mount_file/wd_copy/dwi-preprocessing-using-t1/eddy_fsl/854443cb478db7cdc37ae2446543b7934a6a857d/eddy_fsl/eddy_corrected --repol --slm=none

Run the above command directly in the terminal, I got the error:

Traceback (most recent call last):
  File "/home/local/PARTNERS/fsl/bin/eddy", line 98, in <module>
    main()
  File "/home/local/PARTNERS/fsl/bin/eddy", line 79, in main
    eddy_cuda   = find_suitable_eddy_cuda()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/local/PARTNERS/fsl/bin/eddy", line 58, in find_suitable_eddy_cuda
    eddy_cudas = find_all_eddy_cudas()
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/home/local/PARTNERS/fsl/bin/eddy", line 48, in find_all_eddy_cudas
    eddy_vers  = [float(op.basename(f)[9:]) for f in eddy_cudas]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/local/PARTNERS/fsl/bin/eddy", line 48, in <listcomp>
    eddy_vers  = [float(op.basename(f)[9:]) for f in eddy_cudas]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: could not convert string to float: ''

Replace eddy with eddy_cpu:

eddy_cpu --flm=quadratic --ff=10.0 --acqp=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/acq.txt --bvals=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/sub-ADNI003S4119_ses-M006_dwi_merged.bval --bvecs=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/sub-ADNI003S4119_ses-M006_dwi_merged.bvec --imain=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/sub-ADNI003S4119_ses-M006_dwi_merged.nii.gz --index=/home/local/PARTNERS/mount_file/BIDS_copy/sub-ADNI003S4119/ses-M006/dwi/index.txt --mask=/home/local/PARTNERS/mount_file/wd_copy/dwi-preprocessing-using-t1/eddy_fsl/854443cb478db7cdc37ae2446543b7934a6a857d/mask_reference_b0/reference_b0_volume_brain_mask.nii.gz --interp=spline --resamp=jac --niter=5 --nvoxhp=1000 --out=/home/local/PARTNERS/mount_file/wd_copy/dwi-preprocessing-using-t1/eddy_fsl/854443cb478db7cdc37ae2446543b7934a6a857d/eddy_fsl/eddy_corrected --repol --slm=none

This revised command works.

ghisvail commented 8 months ago

A few questions / remarks:

The most recent version of FSL (The conda-based version installed by the fslinstaller) features a different entry-point script which does not have this bug.

ghisvail commented 8 months ago

Comparing the old to the new entry-point.

felicityshaw commented 7 months ago

Indeed, I noticed that the version of FSL installed was outdated, and I was using Python version 3.10. After reinstalling FSL with the latest fslinstaller.py available on the FSL website, I attempted the process again but encountered the same error as before.

Besides, when I ran the 't1-freesurfer' pipeline by the following command:

clinica run t1-freesurfer -tsv ./tsv/sub-ADNI127S5067.tsv -wd wd_copy BIDS_copy CAPS_copy

I got the error traceback like this: 

Traceback (most recent call last):
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/bin/clinica", line 8, in <module>
    sys.exit(main())
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/pipelines/t1_freesurfer/t1_freesurfer_cli.py", line 73, in cli
    pipeline.run(plugin="MultiProc", plugin_args={"n_procs": n_procs})
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/pipelines/engine.py", line 353, in run
    raise e
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/clinica/pipelines/engine.py", line 334, in run
    exec_graph = Workflow.run(self, plugin, plugin_args, update_hash)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 166, in run
    self._clean_queue(jobid, graph, result=result)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 244, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 524, in run
    result = self._run_interface(execute=True)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 642, in _run_interface
    return self._run_command(execute)
  File "/home/local/PARTNERS/anaconda3/envs/clinicaEnv/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 750, in _run_command
    raise NodeExecutionError(
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node 1-SegmentationReconAll.

RuntimeError: subprocess exited with code 1.

I tried adjusting the n_procs value (to 16, 8, 4, and 1) in an attempt to resolve the issue, yet the problem persisted. Are there specific CPU resource requirements that "nipype" or "multiprocs" necessitates?

NicolasGensollen commented 7 months ago

Hi @felicityshaw

The issue is starting to be difficult to follow. The initial post is about an issue in the DWIPreprocessingUsingT1 pipeline which seems to be linked with the installed version of FSL. Reading your message above, I understand that you were able to install a newer version of FSL. What version did you install ? When you run the DWI pipeline, what error are you getting now ? The exact same one as before ? What happens when you run manually the eddy command in your terminal ?

For the problem with the T1Freesurfer pipeline, I don't think it is related since the underlying executable is the FreeSurfer recon-all. What version of Freesurfer do you have installed on your system ? Are you able to run things like recon-all --help in your terminal ? If yes, could you locate the command which crashed the pipeline and try to run it directly ?

ta4218 commented 6 months ago

I am experiencing an identical error to the initial issue raised. I am working with the development version of Clinica, Python 3.10. My OS is Red Hat Enterprise Linux 8.5 (Ootpa). I was initially using FSL 6.0.0, but have since upgraded to 6.0.7 and I am still experiencing the same error.

The following is the crash file, that occurs for most dwi images when I run the command: clinica run dwi-preprocessing-using-t1 $EPHEMERAL/ADNI234_2/adni_bids_t1 $EPHEMERAL/ADNI234_2/adni_caps_final -tsv $EPHEMERAL/ADNI234_2/missing_modalities_dwi/batches_dwi_updated/dwi_session_batch_0.tsv --n_procs 256 -wd $EPHEMERAL/WD

File: /rds/general/user/ta4218/home/crash-20240326-133211-ta4218-eddy_fsl.a00-5ebb3813-541b-4869-b1ee-dbbc83d53dab.pklz
Node: dwi-preprocessing-using-t1.eddy_fsl.eddy_fsl
Working directory: /rds/general/ephemeral/user/ta4218/ephemeral/WD/dwi-preprocessing-using-t1/eddy_fsl/6d6a937882cc95c3dcdbc8b05e1c6d105ddabfc4/eddy_fsl

Node inputs:

args = <undefined>
cnr_maps = <undefined>
dont_peas = <undefined>
dont_sep_offs_move = <undefined>
environ = {'FSLOUTPUTTYPE': 'NIFTI_GZ', 'OMP_NUM_THREADS': '1'}
estimate_move_by_susceptibility = <undefined>
fep = <undefined>
field = <undefined>
field_mat = <undefined>
flm = quadratic
fudge_factor = 10.0
fwhm = <undefined>
in_acqp = <undefined>
in_bval = <undefined>
in_bvec = <undefined>
in_file = <undefined>
in_index = <undefined>
in_mask = <undefined>
in_topup_fieldcoef = <undefined>
in_topup_movpar = <undefined>
initrand = False
interp = spline
is_shelled = <undefined>
json = <undefined>
mbs_ksp = <undefined>
mbs_lambda = <undefined>
mbs_niter = <undefined>
method = jac
mporder = <undefined>
multiband_factor = <undefined>
multiband_offset = <undefined>
niter = 5
num_threads = 1
nvoxhp = 1000
out_base = eddy_corrected
outlier_nstd = <undefined>
outlier_nvox = <undefined>
outlier_pos = <undefined>
outlier_sqr = <undefined>
outlier_type = <undefined>
output_type = NIFTI_GZ
repol = True
residuals = <undefined>
session = <undefined>
slice2vol_interp = <undefined>
slice2vol_lambda = <undefined>
slice2vol_niter = <undefined>
slice_order = <undefined>
slm = none
use_cuda = False

Traceback: 
Traceback (most recent call last):
  File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy_fsl.

Cmdline:
    eddy --flm=quadratic --ff=10.0 --acqp=/rds/general/ephemeral/user/ta4218/ephemeral/WD/acq.txt --bvals=/rds/general/ephemeral/user/ta4218/ephemeral/WD/5fd786f07e56f68d606d58f9e5b29ab6/sub-ADNI068S0127_ses-M174_dwi_merged.bval --bvecs=/rds/general/ephemeral/user/ta4218/ephemeral/WD/5fd786f07e56f68d606d58f9e5b29ab6/sub-ADNI068S0127_ses-M174_dwi_merged.bvec --imain=/rds/general/ephemeral/user/ta4218/ephemeral/WD/5fd786f07e56f68d606d58f9e5b29ab6/sub-ADNI068S0127_ses-M174_dwi_merged.nii.gz --index=/rds/general/ephemeral/user/ta4218/ephemeral/WD/index.txt --mask=/rds/general/ephemeral/user/ta4218/ephemeral/WD/dwi-preprocessing-using-t1/eddy_fsl/6d6a937882cc95c3dcdbc8b05e1c6d105ddabfc4/mask_reference_b0/reference_b0_volume_brain_mask.nii.gz --interp=spline --resamp=jac --niter=5 --nvoxhp=1000 --out=/rds/general/ephemeral/user/ta4218/ephemeral/WD/dwi-preprocessing-using-t1/eddy_fsl/6d6a937882cc95c3dcdbc8b05e1c6d105ddabfc4/eddy_fsl/eddy_corrected --repol --slm=none
Stdout:
    ��XEV
    Terminating program
Stderr:

Traceback:
    Traceback (most recent call last):
      File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
        setattr(outputs, key, val)
      File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
        value = super(File, self).validate(objekt, name, value, return_pathlike=True)
      File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
        self.error(objekt, name, str(value))
      File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
        raise TraitError(
    traits.trait_errors.TraitError: The 'out_corrected' trait of an EddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/rds/general/ephemeral/user/ta4218/ephemeral/WD/dwi-preprocessing-using-t1/eddy_fsl/6d6a937882cc95c3dcdbc8b05e1c6d105ddabfc4/eddy_fsl/eddy_corrected.nii.gz' <class 'str'> was specified.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
        outputs = self.aggregate_outputs(runtime)
      File "/rds/general/user/ta4218/home/anaconda3/envs/clinicaDev/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
        raise FileNotFoundError(msg)
    FileNotFoundError: No such file or directory '/rds/general/ephemeral/user/ta4218/ephemeral/WD/dwi-preprocessing-using-t1/eddy_fsl/6d6a937882cc95c3dcdbc8b05e1c6d105ddabfc4/eddy_fsl/eddy_corrected.nii.gz' for output 'out_corrected' of a Eddy interface

The input BIDs folder is shown here below:

├── anat │ └── sub-ADNI068S0127_ses-M174_T1w.nii.gz ├── dwi │ ├── reference_b0_volume.nii.gz │ ├── sub-ADNI068S0127_ses-M174_dwi.bval │ ├── sub-ADNI068S0127_ses-M174_dwi.bvec │ ├── sub-ADNI068S0127_ses-M174_dwi.json │ ├── sub-ADNI068S0127_ses-M174_dwi_large_b.bval │ ├── sub-ADNI068S0127_ses-M174_dwi_large_b.bvec │ ├── sub-ADNI068S0127_ses-M174_dwi_large_b.nii.gz │ ├── sub-ADNI068S0127_ses-M174_dwi_merged.bval │ ├── sub-ADNI068S0127_ses-M174_dwi_merged.bvec │ ├── sub-ADNI068S0127_ses-M174_dwi_merged.nii.gz │ ├── sub-ADNI068S0127_ses-M174_dwi.nii.gz │ ├── sub-ADNI068S0127_ses-M174_dwi_small_b.bval │ ├── sub-ADNI068S0127_ses-M174_dwi_small_b.bvec │ └── sub-ADNI068S0127_ses-M174_dwi_small_b.nii.gz └── sub-ADNI068S0127_ses-M174_scans.tsv

If you have any ideas about this error- it would be great to know what is going on?

NicolasGensollen commented 6 months ago

Hi @ta4218

Thanks for reporting. First of all, what version of Clinica are you using ? If in the 0.7.X series, it might be worth trying with the recently released v.0.8.0 which changed a few things in the DWI pipelines.

Moreover, have you tried running the eddy command in your terminal ? Have you checked that the different input files exist ? Have you checked in the full traceback if you had an error before the eddy_fsl node (which could explain that some expected files by downstream nodes are not present) ?

github-actions[bot] commented 3 months ago

This issue is considered stale because it has not received further activity for the last 14 days. You may remove the inactive label or add a comment, otherwise it will be closed after the next 14 days.