🐛 [User-reported Bug] Crash when running a new functional dataset using a shared output directory from previous run

[tbweng]

Describe the bug

Pipeline builder crashes when trying to preprocess a new functional dataset and a previously run functional dataset already exists in the output directory.

To Reproduce Steps to reproduce the behavior:

1. Preprocess a functional dataset (e.g., resting state).
2. Preprocess a different functional dataset (e.g., n-back task) in a new cpac run with shared output directory.
3. See error -->

   Traceback (most recent call last):
   File "/code/run.py", line 677, in <module>
   test_config = 1 if args.analysis_level == "test_config" else 0
   File "/code/CPAC/pipeline/cpac_runner.py", line 549, in run
   p_name, plugin, plugin_args, test_config)
   File "/code/CPAC/pipeline/cpac_pipeline.py", line 359, in run_workflow
   subject_id, sub_dict, c, p_name, num_ants_cores
   File "/code/CPAC/pipeline/cpac_pipeline.py", line 1173, in build_workflow
   rpool.gather_pipes(wf, cfg)
   File "/code/CPAC/pipeline/engine.py", line 978, in gather_pipes
   node, out = self.rpool['fwhm'][fwhm_idx]['data']
   KeyError: "[[[[['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg'], [['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni'], 'desc-brain_bold:bold_masking', 'desc-brain_bold:func_normalize'], [[['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg'], [['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni'], 'desc-brain_bold:bold_masking', 'desc-mean_bold:func_mean'], [[[['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg'], [['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni'], 'desc-brain_bold:bold_masking', 'desc-mean_bold:func_mean'], [[[['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg'], [['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni'], 'desc-brain_bold:bold_masking', 'desc-mean_bold:func_mean', 'desc-reginput_bold:coregistration_prep_mean'], [['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w_template:template_resample'], ['T1w_brain_template:template_resample'], ['T1w_brain_template_mask:T1w_brain_template_mask_config_ingress'], 'from-template_to-T1w_mode-image_desc-linear_xfm:register_ANTs_anat_to_template'], ['CSF_path:CSF_path_config_ingress'], ['GM_path:GM_path_config_ingress'], ['WM_path:WM_path_config_ingress'], 'label-WM_probseg:tissue_seg_fsl_fast'], ['T1w:anat_ingress'], 'from-bold_to-T1w_mode-image_desc-linear_xfm:coregistration'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w_template:template_resample'], ['T1w_brain_template:template_resample'], ['T1w_brain_template_mask:T1w_brain_template_mask_config_ingress'], 'from-T1w_to-template_mode-image_xfm:register_ANTs_anat_to_template'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w_template:template_resample'], ['T1w_brain_template:template_resample'], ['T1w_brain_template_mask:T1w_brain_template_mask_config_ingress'], 'from-template_to-T1w_mode-image_xfm:register_ANTs_anat_to_template'], [['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], ['T1w_brain_template_funcreg:template_resample'], 'from-bold_to-template_mode-image_xfm:create_func_to_T1template_xfm'], [[[['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg'], [['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni'], 'desc-brain_bold:bold_masking', 'desc-mean_bold:func_mean'], [[[['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg'], [['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni'], 'desc-brain_bold:bold_masking', 'desc-mean_bold:func_mean', 'desc-reginput_bold:coregistration_prep_mean'], [['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_mask_afni'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w_template:template_resample'], ['T1w_brain_template:template_resample'], ['T1w_brain_template_mask:T1w_brain_template_mask_config_ingress'], 'from-template_to-T1w_mode-image_desc-linear_xfm:register_ANTs_anat_to_template'], ['CSF_path:CSF_path_config_ingress'], ['GM_path:GM_path_config_ingress'], ['WM_path:WM_path_config_ingress'], 'label-WM_probseg:tissue_seg_fsl_fast'], ['T1w:anat_ingress'], 'from-bold_to-T1w_mode-image_desc-linear_xfm:coregistration'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w_template:template_resample'], ['T1w_brain_template:template_resample'], ['T1w_brain_template_mask:T1w_brain_template_mask_config_ingress'], 'from-T1w_to-template_mode-image_xfm:register_ANTs_anat_to_template'], [[['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w_template:template_resample'], ['T1w_brain_template:template_resample'], ['T1w_brain_template_mask:T1w_brain_template_mask_config_ingress'], 'from-template_to-T1w_mode-image_xfm:register_ANTs_anat_to_template'], [['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init'], ['T1w:anat_ingress', 'desc-preproc_T1w:anatomical_init', 'space-T1w_desc-brain_mask:brain_extraction'], 'desc-brain_T1w:brain_extraction'], ['T1w_brain_template_funcreg:template_resample'], 'from-template_to-bold_mode-image_xfm:create_func_to_T1template_xfm'], ['T1w_brain_template_funcreg:template_resample'], 'desc-cleaned_bold:ICA_AROMA_ANTsreg', 'desc-cleaned_bold:nuisance_regression_defaultNoGSR'], [['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni', 'space-bold_desc-brain_mask:func_normalize'], 'desc-sm_alff:spatial_smoothing_FSL']"

Expected behavior Pull in existing preprocessed anatomical data previous cpac run and only run preprocessing procedures unique to the new functional data.

Versions

• C-PAC: 1.8.0
• Container Platform :
  • Singularity: 3.7.0-1.el7

[shnizzedy]

Related to #1345

[shnizzedy]

@tbweng are you using the same pipeline config for both (attempted) runs here?

[tbweng]

This was happening while using 2 different pipeline configs. My workaround was to 'overwrite' the container's default pipeline with my custom pipeline using singularity -B, e.g.,: singularity run -e -B /cbica/home/wengt/cpac_pipeline_config_RESTING.yml:/cpac_resources/default_pipeline.yml