search

aramis-lab / clinica

Software platform for clinical neuroimaging studies
http://www.clinica.run/
Other
220 stars 74 forks source link

nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node 0-InitNode. #1166

Closed espentrydal closed 2 months ago

espentrydal commented 5 months ago

Describe the bug I still have problems with the dwi-preprocessing-using-t1 pipeline. The pipeline crashes with the below output.

To Reproduce clinica run dwi-preprocessing-using-t1 -tsv ./adni/data/included_subjects_dwi_sorted.tsv -wd ./tmp ./adni/bids ./adni/caps | tee dwi-preprocessing-t1.log

Expected behavior The pipeline to complete successfully.

Output

2024-04-29 02:30:33,734:INFO:Running pipeline for sub-ADNI018S6414 | ses-M000 (TotalReadoutTime = 0.0437881, PhaseEncodingDirection = y)
2024-04-29 02:30:33,744:INFO:Running pipeline for sub-ADNI018S6414 | ses-M012 (TotalReadoutTime = 0.0437129, PhaseEncodingDirection = y)
Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/bin/clinica", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/t1/cli.py", line 69, in cli
    pipeline.run(plugin="MultiProc", plugin_args={"n_procs": n_procs})
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/engine.py", line 715, in run
    raise e
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/engine.py", line 696, in run
    exec_graph = Workflow.run(self, plugin, plugin_args, update_hash)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 178, in run
    self._clean_queue(jobid, graph, result=result)
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 256, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node 0-InitNode.

Traceback:
        Traceback (most recent call last):
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
            out = function_handle(**args)
                  ^^^^^^^^^^^^^^^^^^^^^^^
          File "<string>", line 62, in init_input_node
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/utils.py", line 302, in check_dwi_volume
            num_b_values = len(np.loadtxt(dwi_dataset.b_values))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        TypeError: len() of unsized object

Desktop (please complete the following information):

Additional context All data has been converted to bids-format with clinica v0.8.1. The t1-linear pipeline has also been run successfully with v0.8.1.

Additional logs:

nipypecli show crash-20240429-023035-espen-0-InitNode.a0335-c0a01c88-c1a1-40b6-847e-d1df712d6ec7.pklz

{'node': dwi-preprocessing-using-t1.0-InitNode.a0335,
 'traceback': ['Traceback (most recent call last):\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", '
               'line 67, in run_node\n'
               '    result["result"] = node.run(updatehash=updatehash)\n'
               '                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 527, in run\n'
               '    result = self._run_interface(execute=True)\n'
               '             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 645, in _run_interface\n'
               '    return self._run_command(execute)\n'
               '           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 771, in _run_command\n'
               '    raise NodeExecutionError(msg)\n',
               'nipype.pipeline.engine.nodes.NodeExecutionError: Exception '
               'raised while executing Node 0-InitNode.\n'
               '\n'
               'Traceback:\n'
               '\tTraceback (most recent call last):\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", '
               'line 397, in run\n'
               '\t    runtime = self._run_interface(runtime)\n'
               '\t              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", '
               'line 142, in _run_interface\n'
               '\t    out = function_handle(**args)\n'
               '\t          ^^^^^^^^^^^^^^^^^^^^^^^\n'
               '\t  File "<string>", line 62, in init_input_node\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/utils.py", '
               'line 302, in check_dwi_volume\n'
               '\t    num_b_values = len(np.loadtxt(dwi_dataset.b_values))\n'
               '\t                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
               '\tTypeError: len() of unsized object\n'
               '\n']}
NicolasGensollen commented 5 months ago

Hi @espentrydal

Thanks for reporting those issues ! 👍 I downloaded the subject and try running the DWI preprocessing pipeline and was able to reproduce. It seems like the b-values for the session M-024 is causing some issues:

Session M-000 is fine:

$ cat bids/sub-ADNI018S6414/ses-M000/dwi/sub-ADNI018S6414_ses-M000_dwi.bval
1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0
>>> import numpy as np
>>> np.loadtxt("bids/sub-ADNI018S6414/ses-M000/dwi/sub-ADNI018S6414_ses-M000_dwi.bval")
array([1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000.,
       1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000.,
       1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000.,
       1000., 1000., 1000., 1000., 1000., 1000., 1000., 1000.,    0.])

But session M-024 is not:

$ cat bids/sub-ADNI018S6414/ses-M024/dwi/sub-ADNI018S6414_ses-M024_dwi.bval
0
>>> np.loadtxt("bids/sub-ADNI018S6414/ses-M024/dwi/sub-ADNI018S6414_ses-M024_dwi.bval")
array(0.)
>>> len(np.loadtxt("bids/sub-ADNI018S6414/ses-M024/dwi/sub-ADNI018S6414_ses-M024_dwi.bval"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: len() of unsized object

I need to take a closer look to see what's the best fix.

espentrydal commented 5 months ago

Removing the above image, I ran into some additional problems:

❯cat pypeline.log
240430-13:44:20,735 nipype.utils DEBUG:
         Loading pkl: /home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/2c182bbc15117dca8c7ffb42c4ea0965730ebde8/0-InitNode/result_0-InitNode.pklz
240430-13:44:20,736 nipype.workflow ERROR:
         Node 0-InitNode.a0550 failed to run on host fedora-mainframe.
240430-13:44:20,736 nipype.workflow ERROR:
         Saving crash info to /home/espen/forskningsdata/crash-20240430-134420-espen-0-InitNode.a0550-620a3ed3-2ad7-4a8c-8abd-cdbcd38b474e.pklz
Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node 0-InitNode.

Traceback:
        Traceback (most recent call last):
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
            out = function_handle(**args)
                  ^^^^^^^^^^^^^^^^^^^^^^^
          File "<string>", line 62, in init_input_node
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/utils.py", line 307, in check_dwi_volume
            raise IOError(
        OSError: Number of DWIs, b-vals and b-vecs mismatch (# DWI = 4317, # B-vec = 54, #B-val = 54)
❯ nipypecli show crash-20240430-134420-espen-0-InitNode.a0550-620a3ed3-2ad7-4a8c-8abd-cdbcd38b474e.pklz
{'node': dwi-preprocessing-using-t1.0-InitNode.a0550,
 'traceback': ['Traceback (most recent call last):\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", '
               'line 67, in run_node\n'
               '    result["result"] = node.run(updatehash=updatehash)\n'
               '                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 527, in run\n'
               '    result = self._run_interface(execute=True)\n'
               '             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 645, in _run_interface\n'
               '    return self._run_command(execute)\n'
               '           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 771, in _run_command\n'
               '    raise NodeExecutionError(msg)\n',
               'nipype.pipeline.engine.nodes.NodeExecutionError: Exception '
               'raised while executing Node 0-InitNode.\n'
               '\n'
               'Traceback:\n'
               '\tTraceback (most recent call last):\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", '
               'line 397, in run\n'
               '\t    runtime = self._run_interface(runtime)\n'
               '\t              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", '
               'line 142, in _run_interface\n'
               '\t    out = function_handle(**args)\n'
               '\t          ^^^^^^^^^^^^^^^^^^^^^^^\n'
               '\t  File "<string>", line 62, in init_input_node\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/utils.py", '
               'line 307, in check_dwi_volume\n'
               '\t    raise IOError(\n'
               '\tOSError: Number of DWIs, b-vals and b-vecs mismatch (# DWI = '
               '4317, # B-vec = 54, #B-val = 54) \n'
               '\n']}
❯ cat /home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/2c182bbc15117dca8c7ffb42c4ea0965730ebde8/0-InitNode/_report/report.rst
Node: 0-InitNode (utility)
==========================

 Hierarchy : dwi-preprocessing-using-t1.0-InitNode
 Exec ID : 0-InitNode.a0550

Original Inputs
---------------

* b_values_filename : /home/espen/forskningsdata/adni/bids/sub-ADNI027S6516/ses-M024/dwi/sub-ADNI027S6516_ses-M024_dwi.bval
* b_vectors_filename : /home/espen/forskningsdata/adni/bids/sub-ADNI027S6516/ses-M024/dwi/sub-ADNI027S6516_ses-M024_dwi.bvec
* dwi_filename : /home/espen/forskningsdata/adni/bids/sub-ADNI027S6516/ses-M024/dwi/sub-ADNI027S6516_ses-M024_dwi.nii.gz
* dwi_json_filename : /home/espen/forskningsdata/adni/bids/sub-ADNI027S6516/ses-M024/dwi/sub-ADNI027S6516_ses-M024_dwi.json
* function_str : def init_input_node(
    t1w_filename: str,
    dwi_filename: str,
    dwi_json_filename: str,
    b_vectors_filename: str,
    b_values_filename: str,
) -> tuple:
    """Initialize the pipeline.

    Parameters
    ----------
    t1w_filename : str
        The path to the T1w image in BIDS format.

    dwi_filename : str
        The path to the diffusion weighted image in BIDS format.

    dwi_json_filename : str
        The path to the DWI JSON file in BIDS format and containing
        'TotalReadoutTime' and 'PhaseEncodingDirection' metadata
        (see BIDS specifications).

    b_vectors_filename : str
        The path to the b-vectors file in BIDS format.

    b_values_filename : str
        The path of the b-values file in BIDS format.

    Returns
    -------
    image_id : str
        The subject ID extracted from the t1w image path.

    t1w_filename : str
        The path to the T1w image in BIDS format.

    dwi_filename : str
        The path to the diffusion weighted image in BIDS format.

    b_vectors_filename : str
        The path to the b-vectors file in BIDS format.

    b_values_filename : str
        The path of the b-values file in BIDS format.

    total_readout_time : str
        The total readout time extracted from the dwi JSON file.

    phase_encoding_direction : str
        The phase encoding direction for the dwi image, extracted
        from the dwi JSON file.
    """
    from clinica.pipelines.dwi.preprocessing.utils import (
        check_dwi_volume,
        get_readout_time_and_phase_encoding_direction,
    )
    from clinica.pipelines.dwi.utils import DWIDataset
    from clinica.utils.filemanip import get_subject_id
    from clinica.utils.ux import print_begin_image

    image_id = get_subject_id(t1w_filename)
    check_dwi_volume(
        DWIDataset(
            dwi=dwi_filename,
            b_values=b_values_filename,
            b_vectors=b_vectors_filename,
        )
    )
    (
        total_readout_time,
        phase_encoding_direction,
    ) = get_readout_time_and_phase_encoding_direction(dwi_json_filename)
    print_begin_image(
        image_id,
        ["TotalReadoutTime", "PhaseEncodingDirection"],
        [str(total_readout_time), phase_encoding_direction],
    )

    return (
        image_id,
        t1w_filename,
        dwi_filename,
        b_vectors_filename,
        b_values_filename,
        total_readout_time,
        phase_encoding_direction,
    )

* t1w_filename : /home/espen/forskningsdata/adni/bids/sub-ADNI027S6516/ses-M024/anat/sub-ADNI027S6516_ses-M024_T1w.nii.gz

Similar errors with: sub-ADNI027S6733/ses-M018 sub-ADNI130S4817/ses-M108 sub-ADNI130S5175/ses-M096

espentrydal commented 5 months ago

I also get problems at a later stage in the pipeline (node PrepareB0):

240430-18:02:02,441 nipype.workflow ERROR:
         Node PrepareB0.a0383 failed to run on host fedora-mainframe.
240430-18:02:02,441 nipype.workflow ERROR:
         Saving crash info to /home/espen/forskningsdata/crash-20240430-180202-espen-PrepareB0.a0383-3b3d1b97-1eb1-4fff-83cc-
b72354f08a0a.pklz
Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/bin/clinica", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/t1/cli.py", line 69, in cli
    pipeline.run(plugin="MultiProc", plugin_args={"n_procs": n_procs})
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/engine.py", line 715, in run
    raise e
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/engine.py", line 696, in run
    exec_graph = Workflow.run(self, plugin, plugin_args, update_hash)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 178, in run
    self._clean_queue(jobid, graph, result=result)
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 256, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node PrepareB0.

Traceback:
        Traceback (most recent call last):
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
            out = function_handle(**args)
                  ^^^^^^^^^^^^^^^^^^^^^^^
          File "<string>", line 17, in prepare_reference_b0_task
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/t1/utils.py", line 336, in prepare_reference_b0
            small_b_dataset, large_b_dataset = split_dwi_dataset_with_b_values(
                                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/t1/utils.py", line 687, in split_dwi_dataset_with_b_values
            _build_dwi_dataset_from_filter(
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/t1/utils.py", line 748, in _build_dwi_dataset_from_filter
            dwi=_filter_dwi(
                ^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/t1/utils.py", line 784, in _filter_dwi
            data = compute_aggregated_volume(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/utils/image.py", line 44, in compute_aggregated_volume
            volumes_to_keep = volumes[volumes_to_keep]
                              ~~~~~~~^^^^^^^^^^^^^^^^^
        IndexError: arrays used as indices must be of integer (or boolean) type

Similar errors with following subjects: sub-ADNI018S6414 ses-M024 sub-ADNI020S5203 ses-M078 sub-ADNI020S6504 ses-M024 sub-ADNI027S6516 ses-M024

NicolasGensollen commented 5 months ago

Hi @espentrydal

Sorry for the delay, I started working on this at the beginning of the week, but had to work on other things since then. If you feel confortable enough, you can try to install clinica from this PR: #1169 where I am working on a fix and see whether you can run the DWI pipeline. I'm not sure whether it will solve the issues you're reporting here, but it's worth to try.

If we can come up with a working fix, I'll make a new patch release quickly.

espentrydal commented 4 months ago

Hi @NicolasGensollen , no problem. Thanks!

I installed clinica with git clone --branch fix-dwi-pipeline https://github.com/NicolasGensollen/clinica.git. Unfortunately I still get error messages:

(clinica) ❯ clinica run dwi-preprocessing-using-t1 --use_cuda -tsv ./dwi-test.tsv -wd ./tmp ./adni/bids ./adni/caps 2>&1 | tee dwi-preprocessing-t1.log
2024-05-06 08:35:17,501:INFO:The pipeline will be run on the following 2 image(s):
2024-05-06 08:35:17,501:INFO:   sub-ADNI027S6733 | ses-M000, ses-M018,
2024-05-06 08:35:17,501:INFO:List available in /home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/participants.tsv
2024-05-06 08:35:17,501:INFO:Computational time will depend of the number of volumes in your DWI dataset and the use of CUDA.
Traceback (most recent call last):
  File "/home/espen/miniconda3/envs/clinica/bin/clinica", line 6, in <module>
    sys.exit(main())
  File "/home/espen/src/clinica/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/espen/src/clinica/clinica/pipelines/dwi/preprocessing/t1/cli.py", line 69, in cli
    pipeline.run(plugin="MultiProc", plugin_args={"n_procs": n_procs})
  File "/home/espen/src/clinica/clinica/pipelines/engine.py", line 715, in run
    raise e
  File "/home/espen/src/clinica/clinica/pipelines/engine.py", line 696, in run
    exec_graph = Workflow.run(self, plugin, plugin_args, update_hash)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 178, in run
    self._clean_queue(jobid, graph, result=result)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 256, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node 0-InitNode.

Traceback:
        Traceback (most recent call last):
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
            out = function_handle(**args)
          File "<string>", line 62, in init_input_node
          File "/home/espen/src/clinica/clinica/pipelines/dwi/preprocessing/utils.py", line 307, in check_dwi_volume
            raise IOError(
        OSError: Number of DWIs, b-vals and b-vecs mismatch (# DWI = 1, # B-vec = 54, #B-val = 54)
NicolasGensollen commented 4 months ago

Thanks for tying it out @espentrydal !

I downloaded subject ADNI027S6733 and was able to reproduce. There seems to be a problem with the DWI image for the session M-018:

>>> dwi = nib.load("bids/sub-ADNI027S6733/ses-M018/dwi/sub-ADNI027S6733_ses-M018_dwi.nii.gz")
>>> dwi.shape
(256, 256, 4315)
>>> bvec = np.loadtxt("bids/sub-ADNI027S6733/ses-M018/dwi/sub-ADNI027S6733_ses-M018_dwi.bvec", ndmin=2)
>>> bvec.shape
(3, 54)
>>> bval = np.loadtxt("bids/sub-ADNI027S6733/ses-M018/dwi/sub-ADNI027S6733_ses-M018_dwi.bval", ndmin=1)
>>> bval.shape
(54,)

For comparison, these are the shapes for session M-000:

>>> dwi = nib.load("bids/sub-ADNI027S6733/ses-M000/dwi/sub-ADNI027S6733_ses-M000_dwi.nii.gz")
>>> dwi.shape
(256, 256, 80, 54)

So, clinica is right to raise an error in this case because there is clearly something wrong with this image. However, the pipeline shouldn't crash, instead the error should be handled in order for the problematic image to be skipped.

In #1169 I was trying to make things work even when the DWI image has a single 3D volume (but all shapes match between associated B-vectors and B-values). However, I think those images are the results of a problematic conversion and should clearly be skipped by the DWI preprocessing pipeline (rather than giving results which don't make sense in the end).

I'll modify #1169 to handle those errors rather than accepting wrong images.

I will also try to investigate why those images were not converted properly. It could be a problem with dcm2niix, the DICOM themselves, or the converter code not handling a particular case.

espentrydal commented 4 months ago

Hi @NicolasGensollen

Thanks for investigating! I recently tried again with the fix-dwi-pipeline branch of github.com/NicolasGensollen/clinica with the full set of downloaded subjects.

I get some new error messages, from the eddy_cuda command:

  File "/home/espen/miniconda3/envs/clinica/bin/clinica", line 6, in <module>
    sys.exit(main())
  File "/home/espen/src/clinica/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/espen/src/clinica/clinica/pipelines/dwi/preprocessing/t1/cli.py", line 69, in cli
    pipeline.run(plugin="MultiProc", plugin_args={"n_procs": n_procs})
  File "/home/espen/src/clinica/clinica/pipelines/engine.py", line 719, in run
    raise e
  File "/home/espen/src/clinica/clinica/pipelines/engine.py", line 700, in run
    exec_graph = Workflow.run(self, plugin, plugin_args, update_hash)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 178, in run
    self._clean_queue(jobid, graph, result=result)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 256, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy_fsl.

Cmdline:
        eddy_cuda --flm=quadratic --ff=10.0 --acqp=/home/espen/forskningsdata/tmp/sub-ADNI003S0908_ses-M132_acq.txt --bvals=/home/espen/forskningsdata/tmp/047d8618166e1749b2a1c4aa5492df64/sub-ADNI003S0908_ses-M132_dwi_merged.bval --bvecs=/home/espen/forskningsdata/tmp/047d8618166e1749b2a1c4aa5492df64/sub-ADNI003S0908_ses-M132_dwi_merged.bvec --imain=/home/espen/forskningsdata/tmp/047d8618166e1749b2a1c4aa5492df64/sub-ADNI003S0908_ses-M132_dwi_merged.nii.gz --index=/home/espen/forskningsdata/tmp/sub-ADNI003S0908_ses-M132_index.txt --mask=/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/f51f5b78554496ee7b13787030e3b8d8107af826/mask_reference_b0/reference_b0_volume_brain_mask.nii.gz --interp=spline --resamp=jac --niter=5 --nvoxhp=1000 --out=/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/f51f5b78554496ee7b13787030e3b8d8107af826/eddy_fsl/sub-ADNI003S0908_ses-M132 --repol --slm=none
Stdout:

        Warning: In a future release the first argument will have to be "diffusion" when using eddy on diffusion data, i.e.
        eddy diffusion --imain='my_ima' --acqp='my_acqp' ...

        Warning: Writing of individual text files will be discontinued in favour of a single .json file in future versions

        ...................Allocated GPU # 0...................
        std::bad_alloc: cudaErrorMemoryAllocation: out of memory
        EDDY::cuda/CudaVolume.h:::  EDDY::CudaVolume::CudaVolume(const EDDY::CudaVolume&, bool):  Exception thrown
        EDDY::cuda/EddyInternalGpuUtils.cu:::  static void EDDY::FieldGpuUtils::GetJacobian(const EDDY::CudaVolume4D&, const EDDY::AcqPara&, EDDY::CudaVolume&):  Exception thrown
        EDDY::cuda/EddyInternalGpuUtils.cu:::  static void EDDY::EddyInternalGpuUtils::field_for_scan_to_model_transform(const EDDY::ECScan&, const EDDY::CudaVolume&, EDDY::CudaVolume4D&, EDDY::CudaVolume&, EDDY::CudaVolume&):  Exception thrown
        EDDY::cuda/EddyInternalGpuUtils.cu:::  static void EDDY::EddyInternalGpuUtils::get_unwarped_scan(const EDDY::ECScan&, const EDDY::CudaVolume&, const EDDY::CudaVolume&, const EDDY::CudaVolume&, bool, bool, EDDY::CudaVolume&, EDDY::CudaVolume&):  Exception thrown
        EDDY::cuda/EddyInternalGpuUtils.cu:::  static void EDDY::EddyInternalGpuUtils::load_prediction_maker(const EDDY::EddyCommandLineOptions&, EDDY::ScanType, const EDDY::ECScanManager&, unsigned int, float, bool, std::shared_ptr<EDDY::DWIPredictionMaker>, NEWIMAGE::volume<float>&):  Exception thrown
        EDDY::cuda/EddyGpuUtils.cu:::  static std::shared_ptr<EDDY::DWIPredictionMaker> EDDY::EddyGpuUtils::LoadPredictionMaker(const EDDY::EddyCommandLineOptions&, EDDY::ScanType, const EDDY::ECScanManager&, unsigned int, float, NEWIMAGE::volume<float>&, bool):  Exception thrown
        EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::Register(const EDDY::EddyCommandLineOptions&, EDDY::ScanType, unsigned int, const std::vector<float, std::allocator<float> >&, EDDY::SecondLevelECModelType, bool, EDDY::ECScanManager&, EDDY::ReplacementManager*, NEWMAT::Matrix&, NEWMAT::Matrix&):  Exception thrown
        EDDY::: Eddy failed with message EDDY::eddy.cpp:::  EDDY::ReplacementManager* EDDY::DoVolumeToVolumeRegistration(const EDDY::EddyCommandLineOptions&, EDDY::ECScanManager&):  Exception thrown
Stderr:

Traceback:
        Traceback (most recent call last):
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
            setattr(outputs, key, val)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
            value = super(File, self).validate(objekt, name, value, return_pathlike=True)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
            self.error(objekt, name, str(value))
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
            raise TraitError(
        traits.trait_errors.TraitError: The 'out_parameter' trait of an EddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/f51f5b78554496ee7b13787030e3b8d8107af826/eddy_fsl/sub-ADNI003S0908_ses-M132.eddy_parameters' <class 'str'> was specified.

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
            outputs = self.aggregate_outputs(runtime)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
            raise FileNotFoundError(msg)
        FileNotFoundError: No such file or directory '/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/f51f5b78554496ee7b13787030e3b8d8107af826/eddy_fsl/sub-ADNI003S0908_ses-M132.eddy_parameters' for output 'out_parameter' of a Eddy interface

On an earlier run I got the same error with another subject:

FileNotFoundError: No such file or directory '/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/19390eb91610c8341c949c3f9c69c60727f1d9dc/eddy_fsl/sub-ADNI002S1280_ses-M120.eddy_parameters' for output 'out_parameter' of a Eddy interface

❯ cat $FSLDIR/etc/fslversion
6.0.7.9%

Would this be related to the above errors, or is this something different? Would be grateful for any advice on how to troubleshoot. Thanks!

NicolasGensollen commented 4 months ago

Hi @espentrydal

Sorry for the slow replies here. The FileNotFoundError that you get seems to be a consequence of the error that you have when trying to run eddy_cuda (cudaErrorMemoryAllocation: out of memory). Since the corresponding node fails, the expected output files are missing which triggers this FileNotFoundError. I don't know why you're getting this memory error since I don't know how your system is configured. Were you able to use your GPUs successfully for other subjects/sessions ? Other tasks ?

You could investigate by trying to run the eddy command alone:

eddy_cuda --flm=quadratic --ff=10.0 --acqp=/home/espen/forskningsdata/tmp/sub-ADNI003S0908_ses-M132_acq.txt --bvals=/home/espen/forskningsdata/tmp/047d8618166e1749b2a1c4aa5492df64/sub-ADNI003S0908_ses-M132_dwi_merged.bval --bvecs=/home/espen/forskningsdata/tmp/047d8618166e1749b2a1c4aa5492df64/sub-ADNI003S0908_ses-M132_dwi_merged.bvec --imain=/home/espen/forskningsdata/tmp/047d8618166e1749b2a1c4aa5492df64/sub-ADNI003S0908_ses-M132_dwi_merged.nii.gz --index=/home/espen/forskningsdata/tmp/sub-ADNI003S0908_ses-M132_index.txt --mask=/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/f51f5b78554496ee7b13787030e3b8d8107af826/mask_reference_b0/reference_b0_volume_brain_mask.nii.gz --interp=spline --resamp=jac --niter=5 --nvoxhp=1000 --out=/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/f51f5b78554496ee7b13787030e3b8d8107af826/eddy_fsl/sub-ADNI003S0908_ses-M132 --repol --slm=none

You could try other subjects/sessions or play with the different parameters. Alternatively, you could try running the eddy_openmp command to see how it goes on CPUs.

espentrydal commented 3 months ago

Hi @NicolasGensollen , thanks a lot for your reply. It seems like I tried to many subjects in parallel. Adding the -np option with a low number it seems to work.

But then on some subjects (with -np 1) I get another error:

Traceback (most recent call last):
  File "/home/espen/miniconda3/envs/clinica/bin/clinica", line 6, in <module>
    sys.exit(main())
  File "/home/espen/src/clinica/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "/home/espen/src/clinica/clinica/pipelines/dwi/preprocessing/t1/cli.py", line 69, in cli
    pipeline.run(plugin="MultiProc", plugin_args={"n_procs": n_procs})
  File "/home/espen/src/clinica/clinica/pipelines/engine.py", line 719, in run
    raise e
  File "/home/espen/src/clinica/clinica/pipelines/engine.py", line 700, in run
    exec_graph = Workflow.run(self, plugin, plugin_args, update_hash)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 178, in run
    self._clean_queue(jobid, graph, result=result)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/base.py", line 256, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node eddy_fsl.

Cmdline:
        eddy --flm=quadratic --ff=10.0 --acqp=/home/espen/forskningsdata/tmp/sub-ADNI020S5203_ses-M078_acq.txt --bvals=/home/espen/forskningsdata/tmp/c6b6c9dc8c5115bee96967f49e302372/sub-ADNI020S5203_ses-M078_dwi_small_b.bval --bvecs=/home/espen/forskningsdata/tmp/c6b6c9dc8c5115bee96967f49e302372/sub-ADNI020S5203_ses-M078_dwi_small_b.bvec --imain=/home/espen/forskningsdata/tmp/c6b6c9dc8c5115bee96967f49e302372/sub-ADNI020S5203_ses-M078_dwi_small_b.nii.gz --index=/home/espen/forskningsdata/tmp/sub-ADNI020S5203_ses-M078_index.txt --mask=/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/daff1b0e3f5acdc4e768975a9405d644533cda1b/mask_reference_b0/reference_b0_volume_brain_mask.nii.gz --interp=spline --resamp=jac --niter=5 --nvoxhp=1000 --out=/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/daff1b0e3f5acdc4e768975a9405d644533cda1b/eddy_fsl/sub-ADNI020S5203_ses-M078 --repol --slm=none
Stdout:

        Warning: In a future release the first argument will have to be "diffusion" when using eddy on diffusion data, i.e.
        eddy diffusion --imain='my_ima' --acqp='my_acqp' ...

        Warning: Writing of individual text files will be discontinued in favour of a single .json file in future versions

        EDDY::ECScanManager::GetShellIndicies: Data not shelled
        EDDY::ECScanClasses.cpp:::  std::vector<std::vector<unsigned int> > EDDY::ECScanManager::GetShellIndicies(std::vector<double>&) const:  Exception thrown
        EDDY::: Eddy failed with message EDDY::ECScanClasses.cpp:::  EDDY::ECScanManager::ECScanManager(const string&, const string&, const string&, const string&, const string&, const string&, const string&, const string&, const string&, double, double, EDDY::ECModelType, EDDY::ECModelType, EDDY::LongECModelType, const std::vector<unsigned int>&, const std::vector<unsigned int>&, const EDDY::PolationPara&, EDDY::MultiBandGroups, bool):  Exception thrown
Stderr:

Traceback:
        Traceback (most recent call last):
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
            setattr(outputs, key, val)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
            value = super(File, self).validate(objekt, name, value, return_pathlike=True)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
            self.error(objekt, name, str(value))
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
            raise TraitError(
        traits.trait_errors.TraitError: The 'out_parameter' trait of an EddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/daff1b0e3f5acdc4e768975a9405d644533cda1b/eddy_fsl/sub-ADNI020S5203_ses-M078.eddy_parameters' <class 'str'> was specified.

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
            outputs = self.aggregate_outputs(runtime)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
            raise FileNotFoundError(msg)
        FileNotFoundError: No such file or directory '/home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/eddy_fsl/daff1b0e3f5acdc4e768975a9405d644533cda1b/eddy_fsl/sub-ADNI020S5203_ses-M078.eddy_parameters' for output 'out_parameter' of a Eddy interface

This also happens with sub-ADNI018S6414_ses-M024.

Would you have any idea what might be the cause of this error?

espentrydal commented 3 months ago

Hi @NicolasGensollen ! Just want to check if the most up-to-date branch concerning DWI would still be fix-dwi-pipeline from https://github.com/NicolasGensollen/clinica.git? If that is the case, I will manually try to remove the offending sessions if there are not too many. If there is another branch I should follow instead please let me know. Thanks!

NicolasGensollen commented 3 months ago

Hi @espentrydal ! Sorry for the slow progress on this as I'm busy with other things these days. However, I double-checked the PR #1169 by running the pipeline on subject sub-ADNI027S6733, and I believe it is working as expected (the session M018 is skipped because the DWI image has an unexpected shape):

$ tree bids
bids
├── README
├── dataset_description.json
├── participants.tsv
└── sub-ADNI027S6733
    ├── ses-M000
    │   ├── anat
    │   │   ├── sub-ADNI027S6733_ses-M000_FLAIR.json
    │   │   ├── sub-ADNI027S6733_ses-M000_FLAIR.nii.gz
    │   │   └── sub-ADNI027S6733_ses-M000_T1w.nii.gz
    │   ├── dwi
    │   │   ├── sub-ADNI027S6733_ses-M000_dwi.bval
    │   │   ├── sub-ADNI027S6733_ses-M000_dwi.bvec
    │   │   ├── sub-ADNI027S6733_ses-M000_dwi.json
    │   │   └── sub-ADNI027S6733_ses-M000_dwi.nii.gz
    │   ├── func
    │   │   ├── sub-ADNI027S6733_ses-M000_task-rest_bold.json
    │   │   └── sub-ADNI027S6733_ses-M000_task-rest_bold.nii.gz
    │   ├── pet
    │   │   └── sub-ADNI027S6733_ses-M000_trc-18FFDG_rec-coregiso_pet.nii.gz
    │   └── sub-ADNI027S6733_ses-M000_scans.tsv
    ├── ses-M018
    │   ├── anat
    │   │   ├── sub-ADNI027S6733_ses-M018_FLAIR.json
    │   │   ├── sub-ADNI027S6733_ses-M018_FLAIR.nii.gz
    │   │   └── sub-ADNI027S6733_ses-M018_T1w.nii.gz
    │   ├── dwi
    │   │   ├── sub-ADNI027S6733_ses-M018_dwi.bval
    │   │   ├── sub-ADNI027S6733_ses-M018_dwi.bvec
    │   │   ├── sub-ADNI027S6733_ses-M018_dwi.json
    │   │   └── sub-ADNI027S6733_ses-M018_dwi.nii.gz
    │   ├── func
    │   │   ├── sub-ADNI027S6733_ses-M018_task-rest_bold.json
    │   │   └── sub-ADNI027S6733_ses-M018_task-rest_bold.nii.gz
    │   └── sub-ADNI027S6733_ses-M018_scans.tsv
    └── sub-ADNI027S6733_sessions.tsv
$ clinica run dwi-preprocessing-using-t1 -wd /tmp ./bids/ ./caps
2024-06-20 09:18:11,601:WARNING:Ignoring DWI scan for subject ADNI027S6733 and session M018 for the following reason: Number of DWIs, b-vals and b-vecs mismatch (# DWI = 1, # B-vec = 54, #B-val = 54)
2024-06-20 09:18:12,286:INFO:The pipeline will be run on the following 1 image(s):
2024-06-20 09:18:12,286:INFO:   sub-ADNI027S6733 | ses-M000,
2024-06-20 09:18:12,286:INFO:List available in /tmp/dwi-preprocessing-using-t1/participants.tsv
2024-06-20 09:18:12,286:INFO:Computational time will depend of the number of volumes in your DWI dataset and the use of CUDA.
2024-06-20 17:34:50,869:INFO:sub-ADNI027S6733_ses-M000 has completed
2024-06-20 17:34:56,777:INFO:The dwi-preprocessing-using-t1 pipeline has completed. You can now delete the working directory (/tmp/dwi-preprocessing-using-t1).
$ tree caps
caps
└── subjects
    └── sub-ADNI027S6733
        └── ses-M000
            └── dwi
                └── preprocessing
                    ├── sub-ADNI027S6733_ses-M000_space-T1w_brainmask.nii.gz
                    ├── sub-ADNI027S6733_ses-M000_space-T1w_desc-preproc_dwi.bval
                    ├── sub-ADNI027S6733_ses-M000_space-T1w_desc-preproc_dwi.bvec
                    └── sub-ADNI027S6733_ses-M000_space-T1w_desc-preproc_dwi.nii.gz

I can aim to make a patch release 0.8.2 soonish with this, but you should be able to try it from my branch in the mean time. Let me know if this correctly skips the problematic images or not.

espentrydal commented 3 months ago

Thank you for getting back to me @NicolasGensollen ! No problem, that is very understandable.

The fix does help with ADNI027S6733 and skips it. Unfortunately other subjects give different error messages (on a clean run). This is the case for:

sub-ADNI018S6414 ses-M024 sub-ADNI020S5203 ses-M078 sub-ADNI020S6227 ses-M030 sub-ADNI020S6504 ses-M024 sub-ADNI020S6901 ses-M000

(and possibly more)

With -np in low digits (1-5) I don't seem to get the CUDA out of memory error, so it seems to be because of low memory capacity on my GPU.

Error message (clinica run dwi-preprocessing-using-t1 -np 5 -tsv ./dwi-test.tsv -wd ./tmp-test ./adni/bids ./adni/caps 2>&1 | tee dwi-preprocessing-test.log):

Cmdline:
        eddy --flm=quadratic --ff=10.0 --acqp=/home/espen/forskningsdata/tmp-test/sub-ADNI018S6414_ses-M024_acq.txt --bvals=/home/espen/forskningsdata/tmp-test/dfdcb4aa8949ef465fb2d9955474a1f8/sub-ADNI018S6414_ses-M024_dwi_small_b.bval --bvecs=/home/espen/forskningsdata/tmp-test/dfdcb4aa8949ef465fb2d9955474a1f8/sub-ADNI018S6414_ses-M024_dwi_small_b.bvec --imain=/home/espen/forskningsdata/tmp-test/dfdcb4aa8949ef465fb2d9955474a1f8/sub-ADNI018S6414_ses-M024_dwi_small_b.nii.gz --index=/home/espen/forskningsdata/tmp-test/sub-ADNI018S6414_ses-M024_index.txt --mask=/home/espen/forskningsdata/tmp-test/dwi-preprocessing-using-t1/eddy_fsl/c52a18cf82c7876abd0b16937ce8ef4229cf5c62/mask_reference_b0/sub-ADNI018S6414_ses-M024_dwi_small_b_brain_mask.nii.gz --interp=spline --resamp=jac --niter=5 --nvoxhp=1000 --out=/home/espen/forskningsdata/tmp-test/dwi-preprocessing-using-t1/eddy_fsl/c52a18cf82c7876abd0b16937ce8ef4229cf5c62/eddy_fsl/sub-ADNI018S6414_ses-M024 --repol --slm=none
Stdout:

        Warning: In a future release the first argument will have to be "diffusion" when using eddy on diffusion data, i.e.
        eddy diffusion --imain='my_ima' --acqp='my_acqp' ...

        Warning: Writing of individual text files will be discontinued in favour of a single .json file in future versions

        EddyInputError:  Error when attempting to read --bvecs file /home/espen/forskningsdata/tmp-test/dfdcb4aa8949ef465fb2d9955474a1f8/sub-ADNI018S6414_ses-M024_dwi_small_b.bvec
        Terminating program
Stderr:

Traceback:
        Traceback (most recent call last):
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 453, in aggregate_outputs
            setattr(outputs, key, val)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 330, in validate
            value = super(File, self).validate(objekt, name, value, return_pathlike=True)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/traits_extension.py", line 135, in validate
            self.error(objekt, name, str(value))
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/traits/base_trait_handler.py", line 74, in error
            raise TraitError(
        traits.trait_errors.TraitError: The 'out_parameter' trait of an EddyOutputSpec instance must be a pathlike object or string representing an existing file, but a value of '/home/espen/forskningsdata/tmp-test/dwi-preprocessing-using-t1/eddy_fsl/c52a18cf82c7876abd0b16937ce8ef4229cf5c62/eddy_fsl/sub-ADNI018S6414_ses-M024.eddy_parameters' <class 'str'> was specified.

        During handling of the above exception, another exception occurred:

        Traceback (most recent call last):
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 400, in run
            outputs = self.aggregate_outputs(runtime)
          File "/home/espen/miniconda3/envs/clinica/lib/python3.10/site-packages/nipype/interfaces/base/core.py", line 460, in aggregate_outputs
            raise FileNotFoundError(msg)
        FileNotFoundError: No such file or directory '/home/espen/forskningsdata/tmp-test/dwi-preprocessing-using-t1/eddy_fsl/c52a18cf82c7876abd0b16937ce8ef4229cf5c62/eddy_fsl/sub-ADNI018S6414_ses-M024.eddy_parameters' for output 'out_parameter' of a Eddy interface
❯ cat tmp-test/dfdcb4aa8949ef465fb2d9955474a1f8/sub-ADNI018S6414_ses-M024_dwi_small_b.bvec
   1.00000
   0.00000
   0.00000

EDIT: Got a different error message when rerunning it clean, -np 1, possibly other error message was tainted by a GPU out of memory and then just continuing the run. Added output of .bvec-file.

espentrydal commented 3 months ago 
❯ cat tmp-test/dfdcb4aa8949ef465fb2d9955474a1f8/sub-ADNI018S6414_ses-M024_dwi_small_b.bval
0

That is not the same as in theadni/bids directory:

~/forskningsdata
❯ cat adni/bids/sub-ADNI018S6414/ses-M000/dwi/sub-ADNI018S6414_ses-M000_dwi.bval
1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0

~/forskningsdata
❯ cat adni/bids/sub-ADNI018S6414/ses-M000/dwi/sub-ADNI018S6414_ses-M000_dwi.bvec
0.168631 0.883627 -0.699132 -0.850651 -0.66994 -0.312811 -0.1116 0.536248 -0.0745298 0.913954 -0.15799 0.663848 0.234909 -0.57525 0.850651 0.266101 0.534971 0.849537 -0.852442 -0.27087 0.565761 -0.35597 -0.615092 0.724829 0.16255 -0.922647 -0.66713 -0.525731 -0.997105 0.498602 -0.0892997 0.36072 -0.32338 0.870709 0.18661 0
0.971234 -0.0221299 -0.375421 0.525731 0.717839 -0.519792 -0.68829 -0.11517 0.283379 -0.390772 0.523592 -0.0760798 -0.862075 -0.817901 0.525731 -0.963894 0.727171 0.497638 -0.445111 0.10693 -0.503531 0.55845 0.747523 -0.428669 0.94763 -0.0748197 -0.3008 0 -0.0600297 0.730463 -0.820397 0.33545 0.83462 -0.350429 0.14874 0
0.168131 0.467668 0.608501 0 0.18944 -0.794963 0.7168 0.836167 -0.956107 0.10948 0.837192 -0.743988 0.449048 0.0112 0 -0.00989005 0.43015 -0.175049 -0.274261 0.956659 -0.652971 -0.749279 -0.250741 0.539319 -0.27491 0.378319 -0.68151 -0.850651 -0.0466598 -0.466712 -0.564778 -0.870261 0.44591 -0.345059 0.971109 0

The above is unique in that the values seems like they mean something. Looking at the other sessions that fail, it seems like they do not have correct bval and bvec values in adni/bids directory either (mostly 0s), so it is understandable that they fail.

NicolasGensollen commented 3 months ago

Hi @espentrydal

I downloaded subjects ADNI018S6414 and ADNI020S5203 and I am able to reproduce the error you reported. Indeed, it looks like the bvalues and bvectors of sessions ses-M024 and ses-M078 have some issues:

$ cat bids/sub-ADNI018S6414/ses-M024/dwi/sub-ADNI018S6414_ses-M024_dwi.bval
0
$ cat bids/sub-ADNI018S6414/ses-M024/dwi/sub-ADNI018S6414_ses-M024_dwi.bvec
1
0
0
$ cat bids/sub-ADNI020S5203/ses-M078/dwi/sub-ADNI020S5203_ses-M078_dwi.bval
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
$ cat bids/sub-ADNI020S5203/ses-M078/dwi/sub-ADNI020S5203_ses-M078_dwi.bvec
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

I don't know how these get generated, but this is probably the result of some dcm2niix behavior that was not anticipated.

We should take a closer look on the converter side to avoid producing these images if they don't make sense. In the mean time, I think we should implement a small quality check which would discard images with weird looking bvalues / bvectors such that the DWI pipeline doesn't crash...

espentrydal commented 3 months ago

Thank you @NicolasGensollen for checking it out! That sounds like a good solution.

NicolasGensollen commented 3 months ago

@espentrydal I made a few updates to #1169 in order to skip those problematic images. Feel free to try it, and let me know if you're still getting some errors.

NicolasGensollen commented 2 months ago

This should be fixed in recently released clinica 0.8.2. I'm going to close this, but feel free to re-open if you're still facing related issues.

espentrydal commented 2 months ago

Thanks a lot @NicolasGensollen ! I've been away for some weeks from this part of work, but will try it out as soon as I have the possibility.