search

PennLINC / qsirecon

Reconstruction of preprocessed q-space images (dMRI)
https://qsirecon.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
5 stars 3 forks source link

QSIrecon throwing errors when calculating connectivity from pre-installed atlases and hanging #125

Closed samuelholly closed 3 weeks ago

samuelholly commented 1 month ago

Summary

QSIrecon throwing errors when calculating connectivity from pre-installed atlases and hanging

Additional details

What were you trying to do?

I tried to run the dsi_studio_gqi recon pipeline on my data preprocessed with qsiprep (v 0.19.1).

What did you expect to happen?

That the pipeline will complete succesfully.

What actually happened?

Started getting errors and even after a few hours, there was no progress.

Reproducing the bug

This is the command I ran from my Powershell:

docker run -ti --rm \
    -v H:\qsiprep_out\qsiprep:/data \
    -v H:\qsirecon_out:/out \
    -v W:\license.txt:/opt/freesurfer/license.txt \
    pennlinc/qsirecon:0.23.2 \
    /data /out participant \
    --fs-license-file /opt/freesurfer/license.txt \
    --output-resolution 2 \
    --recon-spec dsi_studio_gqi \
    -vv \
    --n_cpus 8 \
    --resource-monitor \
    --write-graph \
    --notrack \
    --participant-label 019

This is the output in my PS window:

...
241018-11:39:35,87 nipype.workflow WARNING:
         [Node] Error on "dsistudio_atlasgraph.aal116" (/tmp/work/qsirecon_0_23_wf/sub-019_dsistudio_pipeline/sub_019_acq_ep2ddtib3000tra64p2_dir_AP_space_T1w_desc_preproc_recon_wf/streamline_connectivity/calc_connectivity/dsistudio_atlasgraph/aal116)
241018-11:39:37,100 nipype.workflow ERROR:
         Node aal116 failed to run on host cacc95b69241.
241018-11:39:37,229 nipype.workflow ERROR:
         Saving crash info to /tmp/crash-20241018-113937-root-aal116-a5f74bb4-9c50-4fa7-8981-7c93f2772656.pklz
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node aal116.

Cmdline:
        dsi_studio --action=ana  --connectivity=/tmp/work/qsirecon_0_23_wf/sub-019_dsistudio_pipeline/sub_019_acq_ep2ddtib3000tra64p2_dir_AP_space_T1w_desc_preproc_dwi_specific_anat_wf/get_atlases/tpl-MNI152NLin2009cAsym_res-01_atlas-AAL116_desc-LPS_dseg_to_dwi.nii.gz --connectivity_type=pass,end --connectivity_value=count,ncount,mean_length,gfa --source=/tmp/work/qsirecon_0_23_wf/sub-019_dsistudio_pipeline/sub_019_acq_ep2ddtib3000tra64p2_dir_AP_space_T1w_desc_preproc_recon_wf/streamline_connectivity/calc_connectivity/dsistudio_atlasgraph/aal116/sub-019_acq-ep2ddtib3000tra64p2_dir-AP_space-T1w_desc-preproc_dwi.src.gz.odf.gqi.1.25.fib.gz --thread_count=1 --thread_count=1 --tract=/tmp/work/qsirecon_0_23_wf/sub-019_dsistudio_pipeline/sub_019_acq_ep2ddtib3000tra64p2_dir_AP_space_T1w_desc_preproc_recon_wf/streamline_connectivity/calc_connectivity/dsistudio_atlasgraph/aal116/sub-019_acq-ep2ddtib3000tra64p2_dir-AP_space-T1w_desc-preproc_dwi.src.gz.odf.gqi.1.25.fib.trk.gz
Stdout:

Stderr:

Traceback:
        RuntimeError: subprocess exited with code 137.

241018-11:39:37,345 nipype.workflow INFO:
         [Node] Finished "calc_connectivity", elapsed time 672.381246s.
241018-11:39:37,348 nipype.workflow WARNING:
         Storing result file without outputs
...

the same happend with other atlases. Thanks for any help.

tsalo commented 1 month ago

Does the crash file contain any more information?

samuelholly commented 1 month ago

Hi, please see the attachment crash-20241018-113939-root-calc_connectivity-a8780de8-dfa9-4579-bc35-aa391e15d81b.txt

tsalo commented 1 month ago

I think this might end up resolved by #123.

samuelholly commented 3 weeks ago

Hi, thanks for your time. I tried to run the current unstable docker image (as I thought it contains the fix for the original bug) but I got another error, right after processing started - see below:

Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/cli/workflow.py", line 136, in build_workflow
    retval["workflow"] = init_qsirecon_wf()
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/base.py", line 44, in init_qsirecon_wf
    single_subject_wf = init_single_subject_recon_wf(subject_id=subject_id)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/base.py", line 252, in init_single_subject_recon_wf
    dwi_recon_wfs[dwi_file] = init_dwi_recon_workflow(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/recon/build_workflow.py", line 58, in init_dwi_recon_workflow
    new_node = workflow_from_spec(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/recon/build_workflow.py", line 209, in workflow_from_spec
    raise ValueError("Connectivity requires atlases.")
ValueError: Connectivity requires atlases

Thanks in advance for any help. I'm wondering however, how is it possible that other people run qsirecon without problems... Could my out-of-date Docker Desktop be to blame?

tsalo commented 3 weeks ago

What command did you use with the new version?

samuelholly commented 3 weeks ago

this one:

docker run -ti --rm \
-v H:\qsiprep_out\qsiprep:/data \
-v H:\qsirecon_out:/out \
-v W:\license.txt:/opt/freesurfer/license.txt \
pennlinc/qsirecon:unstable /data /out participant \
--fs-license-file /opt/freesurfer/license.txt \ 
--output-resolution 2 \
--recon-spec dsi_studio_gqi \
-vv \
--n_cpus 8 \ 
--resource-monitor \
--write-graph \
--notrack \
--participant-label 019
tsalo commented 3 weeks ago

The new version uses an --atlases parameter instead of defining the atlases in the recon spec. The documentation has information about this new behavior: https://qsirecon.readthedocs.io/en/latest/connectivity_matrices.html#atlases. Can you try updating your command to use the new parameter and re-running?

samuelholly commented 3 weeks ago

Thank you. I tried:

docker run -ti --rm \
-v H:\qsiprep_out\qsiprep:/data \
-v H:\qsirecon_out:/out \
-v W:\license.txt:/opt/freesurfer/license.txt \
pennlinc/qsirecon:unstable /data /out participant \
--fs-license-file /opt/freesurfer/license.txt \
--output-resolution 2 \
--recon-spec dsi_studio_gqi \
-vv --n_cpus 8 \
--resource-monitor \
--write-graph \
--notrack \
--participant-label 019 \
--atlases AAL116

But I'm still getting an error:

Process Process-2:
Traceback (most recent call last):
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/conda/envs/qsiprep/lib/python3.10/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/cli/workflow.py", line 136, in build_workflow
    retval["workflow"] = init_qsirecon_wf()
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/base.py", line 44, in init_qsirecon_wf
    single_subject_wf = init_single_subject_recon_wf(subject_id=subject_id)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/base.py", line 252, in init_single_subject_recon_wf
    dwi_recon_wfs[dwi_file] = init_dwi_recon_workflow(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/recon/build_workflow.py", line 58, in init_dwi_recon_workflow
    new_node = workflow_from_spec(
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/recon/build_workflow.py", line 220, in workflow_from_spec
    return init_dsi_studio_connectivity_wf(**kwargs)
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/qsirecon/workflows/recon/dsi_studio.py", line 539, in init_dsi_studio_connectivity_wf
    workflow.connect([
  File "/opt/conda/envs/qsiprep/lib/python3.10/site-packages/nipype/pipeline/engine/workflows.py", line 178, in connect
    for source, dest in connects:
ValueError: too many values to unpack (expected 2)
tsalo commented 3 weeks ago

That's progress. I think it's a problem with the recon spec then. @mattcieslak @smeisler is this something you could help debug? It doesn't look like the dsi_studio_gqi workflow is being tested in our CI.

mattcieslak commented 3 weeks ago

Back before the split we used to test the dsi_studio_gqi spec as the recon workflow in our preprocess-and-recon test. We should add a standalone test in this repo