Closed sgiavasis closed 4 years ago
sgiavasis
commented
4 years ago @shnizzedy I'd also like to make a plan for when/how to "officially" release the PyPi package. We can fold it into this release if it is stable enough. It doesn't need to be perfect of course.
sgiavasis
commented
4 years ago Current develop/nightly issues I'm nursing along some big testing runs:
I'm going to list issues and errors in this issue. I want to avoid opening new issues for them to avoid confusing other contributors, since these would likely be development/merge errors and won't present in the master/latest.
sgiavasis
commented
4 years ago Still seeing this (got it with regtest-1):
File: /output/crash/crash-20200602-224915-root-sinker_11_slice_time_corrected.b0-8d029d41-9141-4678-a834-770a33f07a25.pklz
Node: resting_preproc_1019436_1.sinker_11_slice_time_corrected
Working directory: /output/working/resting_preproc_1019436_1/_scan_rest_acq-1_run-1/sinker_11_slice_time_corrected
Node inputs:
_outputs = {'slice_time_corrected': '/output/working/resting_preproc_1019436_1/func_slice_timing_correction_0/_scan_rest_acq-1_run-1/slice_timing/sub-1019436_ses-1_task-rest_acq-1_run-1_bold_calc_tshift.nii.gz'}
base_directory = /output/output
bucket = <undefined>
container = pipeline_regtest-1__func-ants_mean_nuisance/1019436_1
creds_path =
encrypt_bucket_keys = False
local_copy = <undefined>
parameterization = True
regexp_substitutions = [('/_sca_roi(.)*[/]', '/'), ('/_smooth_centrality_(\\d)+[/]', '/'), ('/_z_score(\\d)+[/]', '/'), ('/_dr_tempreg_maps_zstat_files_smooth_(\\d)+[/]', '/'), ('/_sca_tempreg_maps_zstat_files_smooth_(\\d)+[/]', '/'), ('/qc___', '/qc/')]
remove_dest_dir = False
strip_dir = <undefined>
substitutions = <undefined>
Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 472, in copyfile
shutil.copyfile(originalfile, newfile)
File "/usr/local/miniconda/lib/python3.7/shutil.py", line 104, in copyfile
raise SameFileError("{!r} and {!r} are the same file".format(src, dst))
shutil.SameFileError: '/output/working/resting_preproc_1019436_1/func_slice_timing_correction_0/_scan_rest_acq-1_run-1/slice_timing/sub-1019436_ses-1_task-rest_acq-1_run-1_bold_calc_tshift.nii.gz' and '/output/output/pipeline_regtest-1__func-ants_mean_nuisance/1019436_1/slice_time_corrected/_scan_rest_acq-1_run-1/sub-1019436_ses-1_task-rest_acq-1_run-1_bold_calc_tshift.nii.gz' are the same file
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
result['result'] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 523, in run
outputs = self.aggregate_outputs(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 597, in aggregate_outputs
predicted_outputs = self._list_outputs()
File "/code/CPAC/utils/interfaces/datasink.py", line 581, in _list_outputs
use_hardlink=use_hardlink)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 474, in copyfile
fmlogger.warn(e.message)
AttributeError: 'SameFileError' object has no attribute 'message'In regtest-2 and regtest-3, getting this, which is related to the censors.tsv stuff (#1256, #1267 etc.):
File: /output/crash/crash-20200602-230807-root-nuisance_regression.a0.b0-ca1b36ac-6b73-4db5-b951-c4cd416f61b2.pklz
Node: resting_preproc_1019436_1.nuisance_regression_2_1.nuisance_regression
Working directory: /output/working/resting_preproc_1019436_1/nuisance_regression_2_1/_scan_rest_acq-1_run-1/_selector_M-D_C-K-1+1-FD-P0.3/nuisance_regression
Node inputs:
TR = <undefined>
args = <undefined>
automask = <undefined>
bandpass = <undefined>
blur = <undefined>
cenmode = KILL
censor = /output/working/resting_preproc_1019436_1/nuisance_regression_2_1/_scan_rest_acq-1_run-1/find_offending_time_points/censors.tsv
censortr = <undefined>
concat = <undefined>
dsort = <undefined>
environ = {}
in_file = /output/working/resting_preproc_1019436_1/func_preproc_afni_mean_3dvolreg_0/_scan_rest_acq-1_run-1/func_normalize/sub-1019436_ses-1_task-rest_acq-1_run-1_bold_calc_tshift_resample_volreg_calc_maths.nii.gz
mask = /output/working/resting_preproc_1019436_1/func_preproc_afni_mean_3dvolreg_0/func_preproc_afni_mean_3dvolreg_0_skullstrip/_scan_rest_acq-1_run-1/func_get_brain_mask_AFNI/sub-1019436_ses-1_task-rest_acq-1_run-1_bold_calc_tshift_resample_volreg_mask.nii.gz
noblock = <undefined>
norm = False
num_threads = 1
ort = /output/working/resting_preproc_1019436_1/nuisance_regressor_2_1/_scan_rest_acq-1_run-1/_selector_M-D_C-K-1+1-FD-P0.3/build_nuisance_regressors/nuisance_regressors.1D
out_file = residuals.nii.gz
outputtype = NIFTI_GZ
polort = 0
stopband = <undefined>
Traceback:
Traceback (most recent call last):
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 69, in run_node
result['result'] = node.run(updatehash=updatehash)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 471, in run
result = self._run_interface(execute=True)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 555, in _run_interface
return self._run_command(execute)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_command
result = self._interface.run(cwd=outdir)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 521, in run
runtime = self._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/afni/base.py", line 110, in _run_interface
return super(AFNICommandBase, self)._run_interface(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 1033, in _run_interface
self.raise_exception(runtime)
File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 970, in raise_exception
).format(**runtime.dictcopy()))
RuntimeError: Command:
3dTproject -input /output/working/resting_preproc_1019436_1/nuisance_regression_2_1/_scan_rest_acq-1_run-1/_selector_M-D_C-K-1+1-FD-P0.3/nuisance_regression/sub-1019436_ses-1_task-rest_acq-1_run-1_bold_calc_tshift_resample_volreg_calc_maths.nii.gz -cenmode KILL -censor /output/working/resting_preproc_1019436_1/nuisance_regression_2_1/_scan_rest_acq-1_run-1/find_offending_time_points/censors.tsv -mask /output/working/resting_preproc_1019436_1/func_preproc_afni_mean_3dvolreg_0/func_preproc_afni_mean_3dvolreg_0_skullstrip/_scan_rest_acq-1_run-1/func_get_brain_mask_AFNI/sub-1019436_ses-1_task-rest_acq-1_run-1_bold_calc_tshift_resample_volreg_mask.nii.gz -ort /output/working/resting_preproc_1019436_1/nuisance_regressor_2_1/_scan_rest_acq-1_run-1/_selector_M-D_C-K-1+1-FD-P0.3/build_nuisance_regressors/nuisance_regressors.1D -polort 0 -prefix residuals.nii.gz
Standard output:
Standard error:
++ 3dTproject: AFNI version=AFNI_20.1.11 (May 28 2020) [64-bit]
++ Authored by: Cox the Algebraic (Linear)
** ERROR: -censor file is too short (151) for dataset (152)
++ input time points = 152 ; censored = 10 ; remaining = 142
++ Setting up regressors
++ 1 Blocks * 1 polynomials -- 1 polort regressors
+ -- 12 other fixed ort regressors
** FATAL ERROR: Cannot continue after above errors :-( :-( :-( !!
** Program compile date = May 28 2020
Return code: 1With regtest-1, and other pipelines, have sometimes been seeing the pipeline hanging at DVARS:
[Node] Setting-up "resting_preproc_1019436_1.gen_motion_stats_afni_mean_3dvolreg_1.cal_DVARS" in "/output/working/resting_preproc_1019436_1/gen_motion_stats_afni_mean_3dvolreg_1/_scan_rest_acq-1_run-1/cal_DVARS".
200602-23:06:58,883 nipype.workflow INFO:
[Node] Running "cal_DVARS" ("CPAC.utils.interfaces.function.Function")
200602-23:07:00,338 nipype.workflow INFO:
[MultiProc] Running 3 tasks, and 356 jobs ready. Free memory (GB): 11.40/12.00, Free processors: 0/3.
Currently running:
* resting_preproc_1019436_1.gen_motion_stats_afni_mean_3dvolreg_1.cal_DVARS
* resting_preproc_1019436_1.qc_skullstrip_11.montage_skull.montage_a
* resting_preproc_1019436_1.qc_skullstrip_11.montage_skull.montage_sNot an error, but seeing this at the beginning of pipeline connection:
functional_nuisance_residuals_smooth already exists in the resource pool
functional_nuisance_residuals_smooth already exists in the resource pool
functional_nuisance_residuals_smooth already exists in the resource pool
functional_nuisance_residuals_smooth already exists in the resource pool
functional_nuisance_residuals_smooth already exists in the resource pool
functional_nuisance_residuals_smooth already exists in the resource pool3dSkullStrip Singularity libgsl error fixed (WIP- if fixed by release)
This was resolved in the v1.6.2a hotfix that is currently available on Master and fcpindi/c-pac:latest.
Test + Validate:
User docs (w/ flowcharts) done:
Regression tests
Note- the "Regtest configs" are expanded/repurposed benchmark-ANTS, benchmark-FNIRT pipelines that are now being dedicated to regression testing (to avoid departing from original benchmark). They are a work in progress.
Release items: