"OSError: Duplicate node name" when using "--recon-only"

Hi Matt and colleagues,

I am currently trying to concatenate qsiprep preprocessing, and three reconstruction workflows (mrtrix_multishell_msmt_ACT-hsvs, mrtrix_multishell_msmt_pyafq_tractometry, amico_noddi).

For that I perform three consecutive qsiprep runs with a singularity container (qsiprep v 0.18.1) from dockerhub:

preprocessing + mrtrix_multishell_msmt_ACT-hsvs singularity run --cleanenv --userns qsiprep-0.18.1 \ raw_bids out participant \ -w wd \ --participant-label sub-1000221 \ --skip-bids-validation \ --nthreads 8 \ --omp-nthreads 8 \ --recon_input qsiprep \ --stop-on-first-crash \ --freesurfer-input freesurfer \ --output-resolution 2 \ --fs-license-file license.txt \ --recon_spec mrtrix_multishell_msmt_ACT-hsvs \ --verbose
mrtrix_multishell_msmt_pyafq_tractometry + skipping preprocessing with --recon-only singularity run --cleanenv --userns qsiprep-0.18.1 \ raw_bids out participant \ -w wd \ --participant-label sub-1000221 \ --skip-bids-validation \ --nthreads 8 \ --omp-nthreads 8 \ --recon_input qsiprep \ --stop-on-first-crash \ --freesurfer-input freesurfer \ --output-resolution 2 \ --fs-license-file license.txt \ --recon_spec mrtrix_multishell_msmt_pyafq_tractometry \ --recon-only \ --verbose
amico_noddi + skipping preprocessing with --recon-only singularity run --cleanenv --userns qsiprep-0.18.1 \ raw_bids out participant \ -w wd \ --participant-label sub-1000221 \ --skip-bids-validation \ --nthreads 8 \ --omp-nthreads 8 \ --recon_input qsiprep \ --stop-on-first-crash \ --freesurfer-input freesurfer \ --output-resolution 2 \ --fs-license-file license.txt \ --recon_spec amico_noddi \ --recon-only \ --verbose

Step 1 works fine, but the other two fail with the following error

+ singularity run --cleanenv --userns qsiprep-0.18.1 raw_bids out participant -w wd --participant-label sub-1000221 --skip-bids-validation --nthreads 8 --omp-nthreads 8 --recon_input qsiprep --stop-on-first-crash --freesurfer-input freesurfer --output-resolution 2 --fs-license-file license.txt --recon_spec mrtrix_multishell_msmt_pyafq_tractometry --recon-only
WARNING: underlay of /etc/localtime required more than 50 (103) bind mounts
230712-16:52:57,283 nipype.workflow IMPORTANT:

    Running qsirecon version 0.18.1:
      * BIDS dataset path: /home/marvin/test/raw_bids.
      * Participant list: ['1000221'].
      * Run identifier: 20230712-165256_ca2f5f55-09da-49e5-bf21-ae44a6c3c10d.

IMPORTANT:nipype.workflow:
    Running qsirecon version 0.18.1:
      * BIDS dataset path: /home/marvin/test/raw_bids.
      * Participant list: ['1000221'].
      * Run identifier: 20230712-165256_ca2f5f55-09da-49e5-bf21-ae44a6c3c10d.

230712-16:52:57,577 nipype.workflow INFO:
     found ['/home/marvin/test/qsiprep/sub-1000221/ses-2/dwi/sub-1000221_ses-2_acq-AP_space-T1w_desc-preproc_dwi.nii.gz', '/home/marvin/test/qsiprep/sub-1000221/sub-1000221/ses-2/dwi/sub-1000221_ses-2_acq-AP_space-T1w_desc-preproc_dwi.nii.gz', '/home/marvin/test/qsiprep/sub-1000221/sub-1000221/sub-1000221/ses-2/dwi/sub-1000221_ses-2_acq-AP_space-T1w_desc-preproc_dwi.nii.gz'] in /home/marvin/test/qsiprep
INFO:nipype.workflow:found ['/home/marvin/test/qsiprep/sub-1000221/ses-2/dwi/sub-1000221_ses-2_acq-AP_space-T1w_desc-preproc_dwi.nii.gz', '/home/marvin/test/qsiprep/sub-1000221/sub-1000221/ses-2/dwi/sub-1000221_ses-2_acq-AP_space-T1w_desc-preproc_dwi.nii.gz', '/home/marvin/test/qsiprep/sub-1000221/sub-1000221/sub-1000221/ses-2/dwi/sub-1000221_ses-2_acq-AP_space-T1w_desc-preproc_dwi.nii.gz'] in /home/marvin/test/qsiprep
230712-16:52:57,578 nipype.workflow INFO:
     Found usable QSIPrep-preprocessed T1w image and mask.
INFO:nipype.workflow:Found usable QSIPrep-preprocessed T1w image and mask.
230712-16:52:57,579 nipype.workflow INFO:
     Found T1w-to-template transforms
INFO:nipype.workflow:Found T1w-to-template transforms
230712-16:52:57,579 nipype.workflow INFO:
     Freesurfer directory /home/marvin/test/freesurfer/sub-1000221 exists for 1000221
INFO:nipype.workflow:Freesurfer directory /home/marvin/test/freesurfer/sub-1000221 exists for 1000221
230712-16:52:57,581 nipype.workflow INFO:
     Anatomical (T1w) available for recon: {'has_qsiprep_5tt_hsvs': False, 'has_freesurfer_5tt_hsvs': False, 'has_freesurfer': True, 'has_qsiprep_t1w': True, 'has_qsiprep_t1w_transforms': True}
INFO:nipype.workflow:Anatomical (T1w) available for recon: {'has_qsiprep_5tt_hsvs': False, 'has_freesurfer_5tt_hsvs': False, 'has_freesurfer': True, 'has_qsiprep_t1w': True, 'has_qsiprep_t1w_transforms': True}
230712-16:52:57,585 nipype.workflow INFO:
     Transforming ODF ROIs into DWI space for visual report.
INFO:nipype.workflow:Transforming ODF ROIs into DWI space for visual report.
230712-16:52:57,626 nipype.interface INFO:
     Response configuration: {'algorithm': 'dhollander', 'nthreads': 8}
INFO:nipype.interface:Response configuration: {'algorithm': 'dhollander', 'nthreads': 8}
230712-16:52:57,626 nipype.interface INFO:
     Using 8 threads in MRtrix3
INFO:nipype.interface:Using 8 threads in MRtrix3
230712-16:52:57,672 nipype.workflow INFO:
     Transforming ODF ROIs into DWI space for visual report.
INFO:nipype.workflow:Transforming ODF ROIs into DWI space for visual report.
230712-16:52:57,713 nipype.interface INFO:
     Response configuration: {'algorithm': 'dhollander', 'nthreads': 8}
INFO:nipype.interface:Response configuration: {'algorithm': 'dhollander', 'nthreads': 8}
230712-16:52:57,713 nipype.interface INFO:
     Using 8 threads in MRtrix3
INFO:nipype.interface:Using 8 threads in MRtrix3
Process Process-2:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/cli/run.py", line 1180, in build_recon_workflow
    retval['workflow'] = init_qsirecon_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/recon/base.py", line 89, in init_qsirecon_wf
    single_subject_wf = init_single_subject_wf(
  File "/usr/local/miniconda/lib/python3.8/site-packages/qsiprep/workflows/recon/base.py", line 244, in init_single_subject_wf
    workflow.connect([(anat_ingress_wf, dwi_recon_wfs[dwi_file], to_connect)])
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/workflows.py", line 161, in connect
    self._check_nodes(newnodes)
  File "/usr/local/miniconda/lib/python3.8/site-packages/nipype/pipeline/engine/workflows.py", line 769, in _check_nodes
    raise IOError('Duplicate node name "%s" found.' % node.name)
OSError: Duplicate node name "sub_1000221_ses_2_acq_AP_space_T1w_desc_preproc_recon_wf" found.
+ --verbose

I have reproduced the error with qsiprep-0.16.0RC3. I can also reproduce the error on our HPC and locally. The error persists if I wipe the working or output directory. If I remove "--recon-only" from the call, preprocessing reruns and the recon workflow works fine. As the problem persists across version I have the hunch that something is wrong with my command call. However, I currently have no idea how to proceed. Any input would be highly appreciated.

By the way. Is there a more elegant solution to performing the three reconstruction workflows than invoking qsiprep three times? And does mrtrix_multishell_msmt_pyafq_tractometry reperform msmt if I have already performed it during step 1 (qsiprep + mrtrix_multishell_msmt_ACT-hsvs)?

Many thanks!

Best, Marvin

PennLINC / qsirecon

"OSError: Duplicate node name" when using "--recon-only" #26