search

FCP-INDI / cpac_run_logs

Repository for information about C-PAC runs.
1 stars 1 forks source link

C-PAC v1.7.3.dev - upgrade/nipype-1.5.1 #6

Closed shnizzedy closed 3 years ago

shnizzedy commented 3 years ago

C-PAC Run Info

**Description** one subject test of default pipeline with upgraded Nipype (1.5.1) https://github.com/FCP-INDI/C-PAC/pull/1392: the default pipeline on s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS with `--participant_ndx 4`
**Version** replace this with the C-PAC version used
**Container** replace this with the container tag (Docker/Singularity) you used
**System** what system did you run on? local machine (Linux/MacOS)? server/cluster? AWS? Flywheel? etc.
**Data Size** how many TRs is your data? (if BOLD)
**Results** ```Python Error of subject workflow resting_preproc_sub-0026009_ses-1 CPAC run error: Pipeline configuration: analysis Subject workflow: resting_preproc_sub-0026009_ses-1 Elapsed run time (minutes): 1088.9922426184019 Timing information saved in /media/ebs/upgrade_nipype/outputs/2020-11-02-default/log/cpac_individual_timing_analysis.csv System time of start: 2020-11-03 16:15:27 ``` crash files (details in comments below): * `_dp_sca_tempreg_maps` × 1960 × 4 * `_dp_sca_roi_files_to_standard` × 52 × 4 * `check_timeseries.a0.a1` × 4 * `afni_centrality.a0.b0` × 4 * `sinker` × 4

Include:

```BASH cpac \ --tag upgrade_nipype-1.5.1 \ run \ s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS /media/ebs/upgrade_nipype/outputs/2020-11-02-default participant \ --save_working_dir --participant_ndx 4 --mem_gb 12 ```

Developers only:
skull_vis_a skull_vis_s
montage_csf_gm_wm_a montage_csf_gm_wm_s
📎 sorted_node_times.txt
shnizzedy commented 3 years ago

_dp_sca_tempreg_maps

_dp_sca_tempreg_maps_files × 1960
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201102-193110-UID1000-_dp_sca_tempreg_maps_files_10-9f34a6c1-1df6-4bd7-b199-9c9d89a3d515.pklz
Node: _dp_sca_tempreg_maps_files_10
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/dp_sca_tempreg_maps_files_1/mapflow/_dp_sca_tempreg_maps_files_10

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/temporal_regression_sca_1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/rename_maps/mapflow/_rename_maps0/sca_tempreg_z_maps_roi_2001.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
_dp_sca_tempreg_maps_zstat_files_smooth × 1960
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201102-225508-UID1000-_dp_sca_tempreg_maps_zstat_files_smooth_10-b6f4d5a9-91f6-4118-b259-09757a74ecef.pklz
Node: _dp_sca_tempreg_maps_zstat_files_smooth_10
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/dp_sca_tempreg_maps_zstat_files_smooth_1/mapflow/_dp_sca_tempreg_maps_zstat_files_smooth_10

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/sca_tempreg_maps_zstat_files_smooth_multi_1/mapflow/_sca_tempreg_maps_zstat_files_smooth_multi_10/sca_tempreg_z_maps_roi_2001_maths.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
_dp_sca_tempreg_maps_files_smooth × 1960
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201102-225613-UID1000-_dp_sca_tempreg_maps_files_smooth_10-615d0b45-75ea-4434-9e03-71375eeafeca.pklz
Node: _dp_sca_tempreg_maps_files_smooth_10
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/dp_sca_tempreg_maps_files_smooth_1/mapflow/_dp_sca_tempreg_maps_files_smooth_10

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/sca_tempreg_maps_files_smooth_multi_1/mapflow/_sca_tempreg_maps_files_smooth_multi_10/sca_tempreg_z_maps_roi_2001_maths.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
_dp_sca_tempreg_maps_zstat_files × 1960
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201102-192856-UID1000-_dp_sca_tempreg_maps_zstat_files_10-cba253ff-1ce2-41a2-a8a4-be45aa60d82d.pklz
Node: _dp_sca_tempreg_maps_zstat_files_10
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/dp_sca_tempreg_maps_zstat_files_1/mapflow/_dp_sca_tempreg_maps_zstat_files_10

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/temporal_regression_sca_1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/rename_maps_zstat/mapflow/_rename_maps_zstat0/sca_tempreg_z_maps_roi_2001.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
shnizzedy commented 3 years ago

_dp_sca_roi_files_to_standard

_dp_sca_roi_files_to_standard × 52
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201102-221819-UID1000-_dp_sca_roi_files_to_standard_171-aa4bfe74-5196-4f65-9693-22b5df172f44.pklz
Node: _dp_sca_roi_files_to_standard_171
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/dp_sca_roi_files_to_standard_1/mapflow/_dp_sca_roi_files_to_standard_171

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/apply_ants_warp_sca_roi_files_to_standard_mapnode__t1_1/mapflow/_apply_ants_warp_sca_roi_files_to_standard_mapnode__t1_171/sca_ROI_5901_antswarp.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
_dp_sca_roi_files_to_standard_smooth × 52
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201103-033756-UID1000-_dp_sca_roi_files_to_standard_smooth_171-1176f5be-805d-400d-ae67-60a92272c971.pklz
Node: _dp_sca_roi_files_to_standard_smooth_171
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/dp_sca_roi_files_to_standard_smooth_1/mapflow/_dp_sca_roi_files_to_standard_smooth_171

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/sca_roi_files_to_standard_smooth_multi_1/mapflow/_sca_roi_files_to_standard_smooth_multi_171/sca_ROI_5901_antswarp_maths.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
_dp_sca_roi_files_to_standard_smooth_fisher_zstd × 52
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201103-081213-UID1000-_dp_sca_roi_files_to_standard_smooth_fisher_zstd_171-4fc36a58-85c5-4d36-aad7-0b76fd0c41ea.pklz
Node: _dp_sca_roi_files_to_standard_smooth_fisher_zstd_171
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/dp_sca_roi_files_to_standard_smooth_fisher_zstd_1/mapflow/_dp_sca_roi_files_to_standard_smooth_fisher_zstd_171

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/fisher_z_score_std_sca_roi_files_to_standard_smooth_1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/fisher_z_score/mapflow/_fisher_z_score71/sca_ROI_5901_antswarp_maths_fisher_zstd.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
_dp_sca_roi_files_to_standard_fisher_zstd × 52
File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201103-081213-UID1000-_dp_sca_roi_files_to_standard_smooth_fisher_zstd_171-4fc36a58-85c5-4d36-aad7-0b76fd0c41ea.pklz
Node: _dp_sca_roi_files_to_standard_smooth_fisher_zstd_171
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/dp_sca_roi_files_to_standard_smooth_fisher_zstd_1/mapflow/_dp_sca_roi_files_to_standard_smooth_fisher_zstd_171

Node inputs:

function_str = CPAC.qc.utils.drop_percent
measure_file = /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/fisher_z_score_std_sca_roi_files_to_standard_smooth_1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_tt_mask_pad_mask_file_..cpac_templates..tt_mask_pad.nii.gz/_fwhm_4/fisher_z_score/mapflow/_fisher_z_score71/sca_ROI_5901_antswarp_maths_fisher_zstd.nii.gz
percent = 99.999

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/code/CPAC/utils/interfaces/function.py", line 152, in _run_interface
    out = function_handle(**args)
  File "/code/CPAC/qc/utils.py", line 1459, in drop_percent
    max_val = np.percentile(data[data != 0.0], percent)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3707, in percentile
    a, q, axis, out, overwrite_input, interpolation, keepdims)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3826, in _quantile_unchecked
    interpolation=interpolation)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3405, in _ureduce
    r = func(a, **kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3941, in _quantile_ureduce_func
    x1 = take(ap, indices_below, axis=axis) * weights_below
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 189, in take
    return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 56, in _wrapfunc
    return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.
shnizzedy commented 3 years ago

check_timeseries.a0.a1.b0 × 4

File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201102-174137-UID1000-check_timeseries.a0.a1.b0-55fe2e69-c579-485f-81bc-ca7fb6a1fdba.pklz
Node: resting_preproc_sub-0026009_ses-1.temporal_regression_sca_1.check_timeseries
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/temporal_regression_sca_1/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/_mask_CC400_mask_file_..cpac_templates..CC400.nii.gz/check_timeseries

Node inputs:

function_str = def check_ts(in_file):
    import os
    import numpy as np

    if in_file.endswith('.txt'):
        try:
            timepoints, rois = np.loadtxt(in_file).shape
        except ValueError:
            timepoints = np.loadtxt(in_file).shape[0]
            rois = 1
        out_file = in_file
    elif in_file.endswith('.csv') or in_file.endswith('.1D'):
        csv_array = np.genfromtxt(in_file, delimiter=',')
        if np.isnan(csv_array[0][0]):
            csv_array = csv_array[1:]
        timepoints, rois = csv_array.shape
        # multiple regression (fsl_glm) needs this format for -design input
        out_file = os.path.join(os.getcwd(), 
                                os.path.basename(in_file).replace('.csv', '.txt'))
        np.savetxt(out_file, csv_array, delimiter='\t')
    if rois > timepoints:
        message = ('\n\n\n****The number of timepoints (' + str(timepoints)
                   + ') is smaller than the number of ROIs to run ('
                   + str(rois) + ') - therefore the GLM is'
                   + ' underspecified and can\'t run.****\n\n\n')
        print(message)
        raise Exception(message)
    else:
        return out_file

in_file = <undefined>

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/utility/wrappers.py", line 142, in _run_interface
    out = function_handle(**args)
  File "<string>", line 27, in check_ts
Exception: 

****The number of timepoints (251) is smaller than the number of ROIs to run (392) - therefore the GLM is underspecified and can't run.****
shnizzedy commented 3 years ago

afni_centrality.a0.b0 × 4

File: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/crash/crash-20201102-174323-UID1000-afni_centrality.a0.b0-6ef9aa3e-99d1-40a1-a723-8fa26091e210.pklz
Node: resting_preproc_sub-0026009_ses-1.afni_centrality_1_eigenvector.afni_centrality
Working directory: /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/afni_centrality_1_eigenvector/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/afni_centrality

Node inputs:

args = <undefined>
autoclip = <undefined>
automask = <undefined>
environ = {'OMP_NUM_THREADS': '3'}
eps = <undefined>
fecm = <undefined>
full = <undefined>
in_file = <undefined>
mask = <undefined>
max_iter = <undefined>
memory = 1.0
num_threads = 3
out_file = eigenvector_centrality_merged.nii.gz
outputtype = AFNI
polort = <undefined>
scale = <undefined>
shift = <undefined>
sparsity = 0.1
thresh = <undefined>

Traceback: 
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/afni/base.py", line 125, in _run_interface
    runtime, correct_return_codes
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.100000 /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/afni_centrality_1_eigenvector/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/afni_centrality/bandpassed_demeaned_filtered_antswarp_flirt.nii.gz
Standard output:
creating output bucket(eigenvector_centrality_merged.nii.gz)
Standard error:
++ 3dECM: AFNI version=AFNI_20.3.01 (Oct 19 2020) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/afni_centrality_1_eigenvector/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/afni_centrality/bandpassed_demeaned_filtered_antswarp_flirt.nii.gz to FLOAT32
     Warnings of this type will be muted for this session.
     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
++ 49428 voxels in -mask dataset
++ vectim-izing input dataset
++ Detrending with polort = 1
++ Calculating ECM with full method (sparsity=0.100%, thresh=-1.200, scale=1.000, shift=0.000,
max_iter=10000, eps=0.000, binary=0, mem=1023918336)
++ Starting create_sparse_corr_array with a memory allowance of 1023918336
++ The sparse array with 1221539 values will take 37.28 MB of memory (budget = 976.10 MB)
++ Extracting sparse correlation array with threshold = -1.200000 and sparsity = 0.10% (1221539)
++ 3 OpenMP threads started
Looping:0*+ WARNING: Could not allocate a new node!
*+ WARNING: Could not allocate a new node!
*+ WARNING: Could not allocate a new node!
123456789,0123456789,0123456789,0123456789,0123456789,.
** ERROR: Memory budget (976.484619 MB) exceeded, consider using ahigher correlation or lower sparsity threshold
*+ WARNING: Error getting sparse weight array.
++ Calculating ECM with full method (sparsity=0.100%, thresh=-1.200, scale=1.000, shift=0.000,
max_iter=10000, eps=0.000, binary=1, mem=1023127488)
++ Starting create_sparse_corr_array with a memory allowance of 1023127488
++ The sparse array with 1221539 values will take 37.28 MB of memory (budget = 975.35 MB)
++ Extracting sparse correlation array with threshold = -1.200000 and sparsity = 0.10% (1221539)
++ 3 OpenMP threads started
Looping:0*+ WARNING: Could not allocate a new node!
*+ WARNING: Could not allocate a new node!
*+ WARNING: Could not allocate a new node!
123456789,0123456789,0123456789,0123456789,0123456789,.
** ERROR: Memory budget (975.730408 MB) exceeded, consider using ahigher correlation or lower sparsity threshold
*+ WARNING: Error getting sparse weight array.
 + creating output dataset in memory

Fatal Signal 11 (SIGSEGV) received
  3dECM main
 Bottom of Debug Stack
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.100000 /media/ebs/upgrade_nipype/outputs/2020-11-02-default/working/resting_preproc_sub-0026009_ses-1/afni_centrality_1_eigenvector/_scan_rest_run-1/_selector_CSF-2mmE-M_aC-CSF+WM-2mm-DPC5_M-SDB_P-2_BP-B0.01-T0.1/afni_centrality/bandpassed_demeaned_filtered_antswarp_flirt.nii.gz
** AFNI version = AFNI_20.3.01  Compile date = Oct 19 2020
** [[Precompiled binary linux_openmp_64: Oct 19 2020]]
** Program Death **
** If you report this crash to the AFNI message board,
** please copy the error messages EXACTLY, and give
** the command line you used to run the program, and
** any other information needed to repeat the problem.
** You may later be asked to upload data to help debug.
Return code: 1
shnizzedy commented 3 years ago

sinker

https://github.com/FCP-INDI/C-PAC/issues/1260