dwi-preprocessing-using-t1 fail with RuntimeError: Mismatch in number of expected outputs

[espentrydal]

Describe the bug dwi-preprocessing pipeline fails with the below errors. The images have earlier been processed with the adni to bids converter.

To Reproduce clinica run dwi-preprocessing-using-t1 -tsv ./adni/data/test_dwi.tsv ./adni/bids ./adni/caps-dwi-test -wd ./tmp

Expected behavior I expected the pipeline to complete successfully.

Stdout/stderr

2024-04-19 15:57:24,618:INFO:The pipeline will be run on the following 2 image(s):
2024-04-19 15:57:24,618:INFO:   sub-ADNI002S0413 | ses-M132,
2024-04-19 15:57:24,618:INFO:   sub-ADNI002S1155 | ses-M126,
2024-04-19 15:57:24,618:INFO:List available in /home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/participants.tsv
2024-04-19 15:57:24,618:INFO:Computational time will depend of the number of volumes in your DWI dataset and the use of CUDA.
2024-04-19 15:58:50,786:INFO:sub-ADNI002S0413_ses-M132 has completed
Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/bin/clinica", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/cmdline.py", line 92, in main
    cli()
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/dwi/preprocessing/t1/cli.py", line 69, in cli
    pipeline.run(plugin="MultiProc", plugin_args={"n_procs": n_procs})
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/engine.py", line 715, in run
    raise e
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/clinica/pipelines/engine.py", line 696, in run
    exec_graph = Workflow.run(self, plugin, plugin_args, update_hash)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/workflows.py", line 638, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 178, in run
    self._clean_queue(jobid, graph, result=result)
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/base.py", line 256, in _clean_queue
    raise RuntimeError("".join(result["traceback"]))
RuntimeError: Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node rename_into_caps.

Traceback:
        Traceback (most recent call last):
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 147, in _run_interface
            raise RuntimeError("Mismatch in number of expected outputs")
        RuntimeError: Mismatch in number of expected outputs

Desktop (please complete the following information):

• OS: Fedora 39 (same error on Debian-based container)
• 8.0.0 (same error with 7.7.7)
• FSL 6.0.7
• Python 3.11.8

Other logs pypeline.log:

240419-15:58:52,757 nipype.utils DEBUG:
         Loading pkl: /home/espen/forskningsdata/tmp/dwi-preprocessing-using-t1/17193e348755931216a0e71560893ee6046866b7/rename_into_caps/result_rename_into_caps.pklz
240419-15:58:52,757 nipype.workflow ERROR:
         Node rename_into_caps.a0 failed to run on host fedora-mainframe.
240419-15:58:52,757 nipype.workflow ERROR:
         Saving crash info to /home/espen/forskningsdata/crash-20240419-155852-espen-rename_into_caps.a0-12ec6482-1012-4897-8d50-9b813a4ff908.pklz
Traceback (most recent call last):
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node rename_into_caps.

Traceback:
        Traceback (most recent call last):
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", line 397, in run
            runtime = self._run_interface(runtime)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
          File "/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", line 147, in _run_interface
            raise RuntimeError("Mismatch in number of expected outputs")
        RuntimeError: Mismatch in number of expected outputs

❯ nipypecli show crash-20240419-143826-espen-rename_into_caps.a1-176ae1d4-7341-4c36-831c-72abd677330b.pklz

{'node': dwi-preprocessing-using-t1.rename_into_caps.a1,
 'traceback': ['Traceback (most recent call last):\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/plugins/multiproc.py", '
               'line 67, in run_node\n'
               '    result["result"] = node.run(updatehash=updatehash)\n'
               '                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 527, in run\n'
               '    result = self._run_interface(execute=True)\n'
               '             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 645, in _run_interface\n'
               '    return self._run_command(execute)\n'
               '           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n',
               '  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/pipeline/engine/nodes.py", '
               'line 771, in _run_command\n'
               '    raise NodeExecutionError(msg)\n',
               'nipype.pipeline.engine.nodes.NodeExecutionError: Exception '
               'raised while executing Node rename_into_caps.\n'
               '\n'
               'Traceback:\n'
               '\tTraceback (most recent call last):\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/base/core.py", '
               'line 397, in run\n'
               '\t    runtime = self._run_interface(runtime)\n'
               '\t              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n'
               '\t  File '
               '"/home/espen/forskningsdata/clinica/.venv/lib64/python3.11/site-packages/nipype/interfaces/utility/wrappers.py", '
               'line 147, in _run_interface\n'
               '\t    raise RuntimeError("Mismatch in number of expected '
               'outputs")\n'
               '\tRuntimeError: Mismatch in number of expected outputs\n'
               '\n']}

Please advice if there is anything I can do to debug this issue further. Any help much appreciated! Thanks!

[NicolasGensollen]

Hi @espentrydal

Thanks for reporting. I'm not sure what's causing this bug atm. Could you provide the content of the pipeline's input BIDS folder (which should be the output folder of the ADNI to BIDS converter if I followed correctly) ? (the output of tree would do just fine as I'm mostly interested in the names of the files/folders). I'm also wondering whether you converted your dataset to BIDS with the same version of Clinica (0.8.0) or whether this was the result of a previous conversion with an older version of clinica ?

[espentrydal]

Thanks for getting back to me. I've uploaded the output of tree of the bids-folder: bids-folder.txt

The conversion was made with an earlier version, 0.7.x (not sure of the exact version).

[espentrydal]

The -tsv file listing the inputs:

❯ cat test_dwi.tsv

participant_id  session_id
sub-ADNI002S0413        ses-M132
sub-ADNI002S1155        ses-M126

[NicolasGensollen]

Thanks for the information @espentrydal ! 👍

I downloaded sub-ADNI002S0413 again and did the full analysis you're trying to do (adni-to-bids converter and dwi-preproc-using-t1 on session M132).

I first encountered an issue in the converter as it seems like one clinical data CSV file has now a different name than before (DXSUM_PDXCONV_ADNIALL seems to be named DXSUM_PDXCONV now... Did you noticed that too ?). I'll open a dedicated issue tomorrow.

After this, I was able to reproduce the error, which occurs at the end of the DWI preprocessing pipeline. I believe this is a bug I introduced while refactoring the pipeline a few months ago. 😓 I'm not sure why this wasn't caught by our non regression test suite (as this pipeline takes a lot of time and ressources to complete, it tends to fail a lot in our CI, so I wouldn't be surprised if it never finished successfully since I made the modifications...).

I'll work on fixing these two issues tomorrow and will make a patch release afterwards.

[espentrydal]

Thanks a lot @NicolasGensollen !

The data I downloaded uses the old name, so that must be new. But I had to remove a date-suffix that had been added to all clinical data csv-files to make the conversion work.

[NicolasGensollen]

@espentrydal I also believe this is new. The date-suffix issue should have been fixed in clinica 0.7.7.

[NicolasGensollen]

@espentrydal I just made a patch release of Clinica (version 0.8.1) which should fix both issues (the one with the converter and the one with the DWI pipeline). I'll close this but feel free to re-open if you're still facing the issue with 0.8.1.

[espentrydal]

Great, thanks!