CPAC 1.8 running - nuisance problem.

hahaai commented 3 years ago

C-PAC Run Info


Description	Run Default, fmriprep, and XCP (modified from fmriprep option)
Version	1.8
Container	Singularity
System	Uppen Cluster.
Data Size	4 HBN CBIC subjects used in the CPAC regression test. Each subject has several bold images (750, 350 and 250 volumes)
Results	It seems the jobs are hanging there. Nuisance was not done, have some crash files related.

Include:

[ ] Run Command: singularity run -e -B /cbica:/cbica /cbica/projects/RBC/CPACTesting/Pipeline_Timing/cpac_1.8.simg /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/HBNData /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAA504CRN participant --pipeline_file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/cpac_pipeline_1.8_default.yml --skip_bids_validator --n_cpus 8 --save_working_dir --mem_gb 15 --participant_label NDARAA504CRN


(the command-line command you used to start the run)

[x] Pipeline Config (can drag file directly into issue)

pipeline_configs.zip

[ ] Data Config (can drag file directly into issue)

Developers only:

[ ] Default Pipeline Diff:


(print-out from cpac_pipe_diff.py)

[ ] Screenshots of brain extraction and registration wireframe overlays from QC pages (if available):


screenshots here

[ ] Node timing information:


(print-out from callback_log_time_parse.py)

[ ] Extracted time series 1D and nuisance regressors 1D correlations against previous version or some benchmark (can use [corr_two_1D.py](https://github.com/sgiavasis/CPAC_regtest_pack/blob/ma

ster/corr_two_1D.py) but heatmaps or other visualizations are good too):


correlations here

hahaai commented 3 years ago

Here are also a few crash files: nipypecli crash default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz

ile: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz Node: cpacsub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpacsub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path = censor_file_path = csf_summary_file_path = custom_file_paths = function_str = CPAC.nuisance.nuisance.gather_nuisance functional_file_path = global_summary_file_path = grey_matter_summary_file_path = motion_parameters_file_path = selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}} tcompcor_file_path = white_matter_summary_file_path =

Traceback: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run runtime = self._run_interface(runtime) File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface out = function_handle(**args) File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance regressor_length)) ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)I found some crash files in the log folder:

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz Node: cpacsub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpacsub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path = censor_file_path = csf_summary_file_path = custom_file_paths = function_str = CPAC.nuisance.nuisance.gather_nuisance functional_file_path = global_summary_file_path = grey_matter_summary_file_path = motion_parameters_file_path = selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}} tcompcor_file_path = white_matter_summary_file_path =

Traceback: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run runtime = self._run_interface(runtime) File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface out = function_handle(**args) File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance regressor_length)) ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)I found some crash files in the log folder:

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz Node: cpacsub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpacsub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path = censor_file_path = csf_summary_file_path = custom_file_paths = function_str = CPAC.nuisance.nuisance.gather_nuisance functional_file_path = global_summary_file_path = grey_matter_summary_file_path = motion_parameters_file_path = selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}} tcompcor_file_path = white_matter_summary_file_path =

Traceback: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run runtime = self._run_interface(runtime) File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface out = function_handle(**args) File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance regressor_length)) ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)I found some crash files in the log folder:

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-221048-rbc-build_nuisance_regressors.a1-698aaaa1-38dd-4378-b9c0-fe84a0d36356.pklz Node: cpacsub-NDARAB348EWR.nuisance_regressors_default_166.build_nuisance_regressors Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpacsub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/build_nuisance_regressors

Node inputs:

acompcor_file_path = censor_file_path = csf_summary_file_path = custom_file_paths = function_str = CPAC.nuisance.nuisance.gather_nuisance functional_file_path = global_summary_file_path = grey_matter_summary_file_path = motion_parameters_file_path = selector = {'Name': 'default', 'Motion': {'include_delayed': True, 'include_squared': True, 'include_delayed_squared': True}, 'aCompCor': {'summary': {'method': 'DetrendPC', 'components': 5}, 'tissues': ['WhiteMatter', 'CerebrospinalFluid'], 'extraction_resolution': 2}, 'CerebrospinalFluid': {'summary': {'filter': 'Mean', 'method': 'Mean'}, 'extraction_resolution': 2, 'erode_mask': True}, 'GlobalSignal': {'summary': {'filter': 'Mean', 'method': 'Mean'}}, 'PolyOrt': {'degree': 2}, 'Bandpass': {'bottom_frequency': 0.01, 'top_frequency': 0.1}} tcompcor_file_path = white_matter_summary_file_path =

Traceback: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run runtime = self._run_interface(runtime) File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface out = function_handle(**args) File "/code/CPAC/nuisance/nuisance.py", line 198, in gather_nuisance regressor_length)) ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/ICA-AROMA_ANTs_template_to_bold_159/_scan_movieTP/func_concat_ICA-AROMA_ANTs_template_to_bold_159/denoised_func_data_nonaggr_0_trans_tcat.nii.gz (250)

hahaai commented 3 years ago

Another crash file type:

nipypecli crash default_NDARAB348EWR/log/crash-20210211-215342-rbc-chunk_warp_ts_to_T1template_205.a0-8e108e9c-661b-4735-b9f0-7e975af67f2c.pklz

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/log/crash-20210211-215342-rbc-chunk_warp_ts_to_T1template_205.a0-8e108e9c-661b-4735-b9f0-7e975af67f2c.pklz Node: cpacsub-NDARAB348EWR.warp_ts_to_T1template_205.chunk_warp_ts_to_T1template_205 Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpacsub-NDARAB348EWR/warp_ts_to_T1template_205/_scan_movieDM/chunk_warp_ts_to_T1template_205

Node inputs:

func_file = function_str = def chunk_ts(func_file, n_cpus): func_img = nb.load(func_file) trs = func_img.shape[3] chunk = trs/n_cpus TR_ranges = []

for chunk_idx in range(0, n_cpus):
    if chunk_idx == n_cpus - 1:
        TR_ranges.append((int(chunk_idx*chunk), int(trs - 1)))
    else:
        TR_ranges.append((int(chunk_idx*chunk), int((chunk_idx+1)*chunk -1)))
return TR_ranges

n_cpus = 8

Traceback: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run runtime = self._run_interface(runtime) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface out = function_handle(**args) File "", line 3, in chunk_ts IndexError: tuple index out of range

hahaai commented 3 years ago

pipelines used mcflirt (fmriprep and XCP) also has the following crash file:

nipypecli crash fmriprep_NDARAB348EWR/log/crash-20210211-213148-rbc-get_mcflirt_rms_abs_90-c1eee511-cff5-4fc7-b2bc-204c595b2a3f.pklz

File: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/fmriprep_NDARAB348EWR/log/crash-20210211-213148-rbc-get_mcflirt_rms_abs_90-c1eee511-cff5-4fc7-b2bc-204c595b2a3f.pklz Node: cpac__sub-NDARAB348EWR.get_mcflirt_rms_abs_90 Working directory: /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/fmriprep_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/get_mcflirt_rms_abs_90

Node inputs:

function_str = def get_mcflirt_rms_abs(rms_files): for path in rms_files: if 'abs.rms' in path: return path

rms_files =

Traceback: Traceback (most recent call last): File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node result["result"] = node.run(updatehash=updatehash) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run result = self._run_interface(execute=True) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface return self._run_command(execute) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command result = self._interface.run(cwd=outdir) File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run runtime = self._run_interface(runtime) File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface out = function_handle(**args) TypeError: get_mcflirt_rms_abs() missing 1 required positional argument: 'rms_files'

shnizzedy commented 3 years ago

ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file

I wonder if this is a data quality issue or if something isn't mapping correctly. It looks like it's creating 0-length ROI files.

hahaai commented 3 years ago

ValueError: Number of time points in /cbica/projects/RBC/CPACTesting/Pipeline_Timing/Running_problem_testing/AllNew/CPAC_out/default_NDARAB348EWR/working/cpac__sub-NDARAB348EWR/nuisance_regressors_default_166/_scan_movieTP/CerebrospinalFluid_mean/denoised_func_data_nonaggr_0_trans_tcat_flirt_roistat.1D (0) is inconsistent with length of functional file

I wonder if this is a data quality issue or if something isn't mapping correctly. It looks like it's creating 0-length ROI files.

Actually this the lateral ventrical problem where the reference has 3 volume in it rather than 1. This has beee fixed.

FCP-INDI / cpac_run_logs

CPAC 1.8 running - nuisance problem. #9

C-PAC Run Info