Description of output files?

[jlhanson5]

Hello CPAC users,

Is there a data dictionary or descriptions of the CPAC folders/outputs? Right now, I'm processing some resting state data and have the following folders:

afni_centrality_0_degree afni_centrality_0_eigenvector afni_centrality_0_lfcd afni_centrality_1_degree afni_centrality_1_eigenvector afni_centrality_1_lfcd alff_collect_transforms_0 alff_collect_transforms_1 alff_falff_0 alff_fsl_to_itk_0 alff_fsl_to_itk_1 alff_to_standard_0 alff_to_standard_1 anat_gather_0 anat_mni_ants_register_0 anat_preproc_0 anat_symmetric_mni_ants_register_0 apply_ants_warp_functional_brain_mask_to_standard_0 apply_ants_warp_functional_brain_mask_to_standard_1 apply_ants_warp_functional_to_standard_0 apply_ants_warp_functional_to_standard_1 apply_ants_warp_mean_functional_to_standard_0 apply_ants_warp_mean_functional_to_standard_1 apply_ants_warp_motion_correct_to_standard_0 apply_ants_warp_motion_correct_to_standard_1 centrality_zscore_0 centrality_zscore_1 collect_transforms_functional_brain_mask_to_standard_0 collect_transforms_functional_brain_mask_to_standard_1 collect_transforms_functional_to_standard_0 collect_transforms_functional_to_standard_1 collect_transforms_mean_functional_to_standard_0 collect_transforms_mean_functional_to_standard_1 collect_transforms_motion_correct_to_standard_0 collect_transforms_motion_correct_to_standard_1 d3.js dr_tempreg_maps_files_collect_transforms_0 dr_tempreg_maps_files_collect_transforms_1 dr_tempreg_maps_files_fsl_to_itk_0 dr_tempreg_maps_files_fsl_to_itk_1 dr_tempreg_maps_files_to_standard_0 dr_tempreg_maps_files_to_standard_1 dr_tempreg_maps_stack_collect_transforms_0 dr_tempreg_maps_stack_collect_transforms_1 dr_tempreg_maps_stack_fsl_to_itk_0 dr_tempreg_maps_stack_fsl_to_itk_1 dr_tempreg_maps_stack_to_standard_0 dr_tempreg_maps_stack_to_standard_1 dr_tempreg_maps_zstat_files_collect_transforms_0 dr_tempreg_maps_zstat_files_collect_transforms_1 dr_tempreg_maps_zstat_files_fsl_to_itk_0 dr_tempreg_maps_zstat_files_fsl_to_itk_1 dr_tempreg_maps_zstat_files_to_standard_0 dr_tempreg_maps_zstat_files_to_standard_1 dr_tempreg_maps_zstat_stack_collect_transforms_0 dr_tempreg_maps_zstat_stack_collect_transforms_1 dr_tempreg_maps_zstat_stack_fsl_to_itk_0 dr_tempreg_maps_zstat_stack_fsl_to_itk_1 dr_tempreg_maps_zstat_stack_to_standard_0 dr_tempreg_maps_zstat_stack_to_standard_1 edit_func_0 falff_collect_transforms_0 falff_collect_transforms_1 falff_fsl_to_itk_0 falff_fsl_to_itk_1 falff_to_standard_0 falff_to_standard_1 fristons_parameter_model_0 fsl_to_itk_functional_brain_mask_to_standard_0 fsl_to_itk_functional_brain_mask_to_standard_1 fsl_to_itk_functional_to_standard_0 fsl_to_itk_functional_to_standard_1 fsl_to_itk_mean_functional_to_standard_0 fsl_to_itk_mean_functional_to_standard_1 fsl_to_itk_motion_correct_to_standard_0 fsl_to_itk_motion_correct_to_standard_1 func_gather_0 func_preproc_automask_0 func_to_anat_bbreg_0 func_to_anat_FLIRT_0 gen_motion_stats_0 graph1.json graph_detailed.dot graph.dot graph.json index.html log_alff_falff_0 log_alff_to_standard_smooth_0 log_alff_to_standard_smooth_1 log_anat_mni_ants_register_0 log_anat_preproc_0 log_anat_symmetric_mni_ants_register_0 log_apply_ants_warp_functional_brain_mask_to_standard_0 log_apply_ants_warp_functional_brain_mask_to_standard_1 log_apply_ants_warp_functional_to_standard_0 log_apply_ants_warp_functional_to_standard_1 log_apply_ants_warp_mean_functional_to_standard_0 log_apply_ants_warp_mean_functional_to_standard_1 log_apply_ants_warp_motion_correct_to_standard_0 log_apply_ants_warp_motion_correct_to_standard_1 log_dr_tempreg_maps_stack_smooth_0 log_dr_tempreg_maps_stack_smooth_1 log_falff_to_standard_smooth_0 log_falff_to_standard_smooth_1 log_frequency_filter_0 log_fristons_parameter_model_0 log_func_preproc_automask_0 log_gen_motion_stats_0 log_motion_correct_to_standard_smooth_0 log_motion_correct_to_standard_smooth_1 log_network_centrality_smooth_0 log_network_centrality_smooth_1 log_nuisance_0 log_reho_0 log_reho_1 log_reho_to_standard_smooth_0 log_reho_to_standard_smooth_1 log_roi_timeseries_0 log_roi_timeseries_1 log_seg_preproc_0 log_spatial_map_timeseries_0 log_spatial_map_timeseries_1 log_spatial_map_timeseries_for_DR_0 log_spatial_map_timeseries_for_DR_1 log_temporal_dual_regression_0 log_temporal_dual_regression_1 log_vmhc_0 log_vmhc_1 nuisance_0 process_outputs_10 process_outputs_101 process_outputs_102 process_outputs_103 process_outputs_104 process_outputs_105 process_outputs_106 process_outputs_107 process_outputs_108 process_outputs_109 process_outputs_11 process_outputs_110 process_outputs_111 process_outputs_112 process_outputs_113 process_outputs_114 process_outputs_115 process_outputs_12 process_outputs_13 process_outputs_14 process_outputs_15 process_outputs_150 process_outputs_151 process_outputs_16 process_outputs_17 process_outputs_18 process_outputs_183 process_outputs_184 process_outputs_185 process_outputs_186 process_outputs_190 process_outputs_191 process_outputs_4 process_outputs_5 process_outputs_52 process_outputs_53 process_outputs_6 process_outputs_7 process_outputs_8 process_outputs_85 process_outputs_86 process_outputs_87 process_outputs_88 process_outputs_9 process_outputs_92 process_outputs_93 reho_0 reho_1 reho_collect_transforms_0 reho_collect_transforms_1 reho_fsl_to_itk_0 reho_fsl_to_itk_1 reho_to_standard_0 reho_to_standard_1 roi_dataflow_0 roi_dataflow_1 roi_timeseries_0 roi_timeseries_1 _scan_task-movieDM _scan_task-movieTP _scan_task-peer_run-1 _scan_task-peer_run-2 _scan_task-peer_run-3 _scan_task-resting_run-1 _scan_task-resting_run-2 seg_preproc_0 sinker_10 sinker_101 sinker_102 sinker_103 sinker_104 sinker_105 sinker_106 sinker_107 sinker_108 sinker_109 sinker_11 sinker_110 sinker_111 sinker_112 sinker_113 sinker_114 sinker_115 sinker_12 sinker_13 sinker_14 sinker_15 sinker_150 sinker_151 sinker_16 sinker_17 sinker_18 sinker_183 sinker_184 sinker_185 sinker_186 sinker_190 sinker_191 sinker_4 sinker_5 sinker_52 sinker_53 sinker_6 sinker_7 sinker_8 sinker_85 sinker_86 sinker_87 sinker_88 sinker_9 sinker_92 sinker_93 spatial_map_dataflow_0 spatial_map_dataflow_1 spatial_map_dataflow_for_DR_0 spatial_map_dataflow_for_DR_1 spatial_map_timeseries_0 spatial_map_timeseries_1 spatial_map_timeseries_for_DR_0 spatial_map_timeseries_for_DR_1 temporal_dual_regression_0 temporal_dual_regression_1 vmhc_0 vmhc_1

I can determine what most of those outputs are, but I wondered if there was a basic document that outlined them (and how they connected to the measures noted on the first cpac github page, e.g., alff, ReHo, DC, EC, SCA, etc.).

Thanks much, Jamie.

[ccraddock]

Hey Jamie,

I agree that we need to do better defining the outputs, and I will add this document to the list of features to add. In the meantime, I will output an example structure that is annotated. Just please give me a day or two.

We are also working on significantly improving the output directory structure using BIDS derivative specification (when it becomes more solidified).

-cc

[jlhanson5]

CC,

This is great, thanks for this help! There are a LOT of directories, so this is very helpful!

Thanks again, Jamie.

[ccraddock]

are you sure that you are looking in the outputs directory? many of the things that you listed are in the working directory and can be ignored. The outputs directory should be easier to understand. Still working on the translation.

[jlhanson5]

Hmmmm, but of my output folders (pipeline_analysis and pipeline_analysis__freq-filter) are extensive... (see the attached screenshot; the last 'ls' commands show ~80 directories).

[ccraddock]

Yes, but this doesn't include many of the directories in your first post (for example: sinker_87) that are exclusive to the working directory.

I am working on a script to help clean up the output dir, and found this resource which may help you in the meantime: https://github.com/FCP-INDI/C-PAC/blob/a9a7fd811602db8388b548da8eabbc3c6db4eb27/CPAC/GUI/resources/html/_sources/_static/outputs/derivatives.csv

[jlhanson5]

Great, thanks for this help!

A few follow-ups: 1) I don't really see Seed-based Correlation Analysis (SCA) folders (unless I'm missing the specific directory they are in?). 2) There seems to be a fair bit of redundancy. For example, there's two folders in output (pipeline_analysis and pipeline_analysis__freq-filter) which have a fair # of sub-directories. Also, there's additional folders in working. How do these different directories differ? It wasn't clear from that past derivatives file you linked.

Thanks again for the assist!

[ccraddock]

Hey Jamie,

1. SCA isn't turned on in the BIDS apps's default configuration, although it can be easily enabled. This tends to be a more hypothesis driven technique, so we leave it up to the user to define.

2. You are correct about the redundancy and we are working on removing it. They pipeline 'forks' to generate its outputs with and without bandpass filtering. For some reason this results in the creation of separate pipelines one with filtering on and the other with filtering off. Some of the outputs that are not effected by filtering are duplicated between the directories. This is in comparison to other forking that exists in the pipeline, such as with / without global signal regression, that are handled correctly.

-cc

[jlhanson5]

RE: #1, Ah interesting. To clarify (related) outputs, where are the "fully" preprocessed resting state volumes stored (e.g., slice-time and motion corrected; nuisance regressed nifti files)? Is that in "output/pipeline_analysis__freq-filter/sub-NDARAA075AMK_ses-1/functional_to_standard/"? Folks in my group were going to run SCA post CPAC...

RE: #2, in the output folder (derivatives/CPAC_rest/sub-####/output), there's two folders (pipeline_analysis and pipeline_analysis__freq-filter), is it reasonable to just remove pipeline_analysis? It sounds like there's redundancy between the two (just bandpassing in the latter?).

Thanks much for all the assistance! This is VERY helpful!

[ccraddock]

Sorry again for the long wait. In the newer version of CPAC the outputs have been pared down quite a bit to make them more understandable. I have also been working on a script to convert the names to bids format: https://github.com/FCP-INDI/cpac_outputs_to_bids.

for #1, the correct file should be fucntional_to_standard

2, there is a fork in your pipeline where the derivatives are calculated with and without bandpass filtering. This will be handled more clearly named in the future. For now, you should find that the directories are not completely the same. pipeline_analysis should contain some directories, such as anatomical preprocessing, that are not in the pipeline_analysis__freq-filter folder.