search

FCP-INDI / cpac_run_logs

Repository for information about C-PAC runs.
1 stars 1 forks source link

C-PAC v1.8.0.dev - regtest preconfigs #15

Open shnizzedy opened 3 years ago

shnizzedy commented 3 years ago

These all failed quickly and similarly, so I'm combining into one log.

C-PAC Run Info
Description regtest preconfigs
Version v1.8.0.dev
Container Docker version 18.09.7, build 2d0083d
fcpindi/c-pac:dev-v1.8 (62e1dcd1fa33) 2021-02-18T10:00:22.916690611Z
System AWS EC2

regtest-1

Results

Traceback (most recent call last):
  File "/code/run.py", line 662, in <module>
    test_config = 1 if args.analysis_level == "test_config" else 0
  File "/code/CPAC/pipeline/cpac_runner.py", line 549, in run
    p_name, plugin, plugin_args, test_config)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 347, in run_workflow
    subject_id, sub_dict, c, p_name, num_ants_cores
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 1055, in build_workflow
    wf = connect_pipeline(wf, cfg, rpool, pipeline_blocks)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 806, in connect_pipeline
    wf = nb.connect_block(wf, cfg, rpool)
  File "/code/CPAC/pipeline/engine.py", line 1096, in connect_block
    pipe_x, opt)
  File "/code/CPAC/sca/sca.py", line 508, in dual_regression
    f'spatial_map_dataflow_for_DR_{pipe_num}'
  File "/code/CPAC/utils/datasource.py", line 991, in create_spatial_map_dataflow
    spatial_map_dict[base_name]
ValueError: Files with same name not allowed: s3://fcp-indi/resources/cpac/resources/PNAS_Smith09_rsn10.nii.gz /cpac_templates/PNAS_Smith09_rsn10.nii.gz

Include:

cpac -B /media/ebs/CPAC_regtest_pack:/configs /media/ebs/C-PAC:/code /media/ebs/C-PAC/dev/docker_data:/cpac_resources/  \
    --platform docker --tag dev-v1.8 \
    run /media/ebs/runs /media/ebs/runs/2021-02-18/regtest-1 participant \
    --preconfig regtest-1 \
    --save_working_dir \
    --data_config_file /configs/data_config_regtest.yml \
    --n_cpus 8 \
    --mem_gb 15

regtest-2

Results

Process Process-1:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 220, in run_workflow
    c)
  File "/code/CPAC/utils/utils.py", line 1372, in check_config_resources
    raise Exception(err_msg)
Exception: Number of threads for OMP: 1 is greater than the number of threads per subject: 0. Change this and try again.

× 33 with the only difference between each message being the incrementing Process- number (1 ‒ 33)

Include:

cpac -B /media/ebs/CPAC_regtest_pack:/configs /media/ebs/C-PAC:/code /media/ebs/C-PAC/dev/docker_data:/cpac_resources/  \
    --platform docker --tag dev-v1.8 \
    run /media/ebs/runs /media/ebs/runs/2021-02-18/regtest-2 participant \
    --preconfig regtest-2 \
    --save_working_dir \
    --data_config_file /configs/data_config_regtest.yml \
    --n_cpus 8 \
    --mem_gb 15

regtest-3

Results

Process Process-1:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 220, in run_workflow
    c)
  File "/code/CPAC/utils/utils.py", line 1372, in check_config_resources
    raise Exception(err_msg)
Exception: Number of threads for OMP: 1 is greater than the number of threads per subject: 0. Change this and try again.

× 31 with the only difference between each message being the incrementing Process- number (1 ‒ 31)

Include:

cpac -B /media/ebs/CPAC_regtest_pack:/configs /media/ebs/C-PAC:/code /media/ebs/C-PAC/dev/docker_data:/cpac_resources/  \
    --platform docker --tag dev-v1.8 \
    run /media/ebs/runs /media/ebs/runs/2021-02-18/regtest-3 participant \
    --preconfig regtest-3 \
    --save_working_dir \
    --data_config_file /configs/data_config_regtest.yml \
    --n_cpus 8 \
    --mem_gb 15

regtest-4

Results

Process Process-1:
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/usr/local/miniconda/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 220, in run_workflow
    c)
  File "/code/CPAC/utils/utils.py", line 1372, in check_config_resources
    raise Exception(err_msg)
Exception: Number of threads for OMP: 1 is greater than the number of threads per subject: 0. Change this and try again.

× 32 with the only difference between each message being the incrementing Process- number (1 ‒ 32)

Traceback (most recent call last):
  File "/code/run.py", line 662, in <module>
    test_config = 1 if args.analysis_level == "test_config" else 0
  File "/code/CPAC/pipeline/cpac_runner.py", line 597, in run
    processes[idx].start()
IndexError: list index out of range

Include:

cpac -B /media/ebs/CPAC_regtest_pack:/configs /media/ebs/C-PAC:/code /media/ebs/C-PAC/dev/docker_data:/cpac_resources/  \
    --platform docker --tag dev-v1.8 \
    run /media/ebs/runs /media/ebs/runs/2021-02-18/regtest-4 participant \
    --preconfig regtest-4 \
    --save_working_dir \
    --data_config_file /configs/data_config_regtest.yml \
    --n_cpus 8 \
    --mem_gb 15
shnizzedy commented 3 years ago

regtest-2 with https://github.com/FCP-INDI/C-PAC/pull/1444

Traceback (most recent call last):
  File "/code/run.py", line 677, in <module>
    test_config = 1 if args.analysis_level == "test_config" else 0
  File "/code/CPAC/pipeline/cpac_runner.py", line 549, in run
    p_name, plugin, plugin_args, test_config)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 347, in run_workflow
    subject_id, sub_dict, c, p_name, num_ants_cores
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 1066, in build_workflow
    wf = connect_pipeline(wf, cfg, rpool, pipeline_blocks)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 806, in connect_pipeline
    wf = nb.connect_block(wf, cfg, rpool)
  File "/code/CPAC/pipeline/engine.py", line 1097, in connect_block
    pipe_x, opt)
  File "/code/CPAC/sca/sca.py", line 505, in dual_regression
    f'spatial_map_dataflow_for_DR_{pipe_num}'
  File "/code/CPAC/utils/datasource.py", line 991, in create_spatial_map_dataflow
    spatial_map_dict[base_name]
ValueError: Files with same name not allowed: s3://fcp-indi/resources/cpac/resources/PNAS_Smith09_rsn10.nii.gz /cpac_templates/PNAS_Smith09_rsn10.nii.gz
shnizzedy commented 3 years ago

regtest-3 with https://github.com/FCP-INDI/C-PAC/pull/1444

Traceback (most recent call last):
  File "/code/run.py", line 677, in <module>
    test_config = 1 if args.analysis_level == "test_config" else 0
  File "/code/CPAC/pipeline/cpac_runner.py", line 549, in run
    p_name, plugin, plugin_args, test_config)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 347, in run_workflow
    subject_id, sub_dict, c, p_name, num_ants_cores
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 1066, in build_workflow
    wf = connect_pipeline(wf, cfg, rpool, pipeline_blocks)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 806, in connect_pipeline
    wf = nb.connect_block(wf, cfg, rpool)
  File "/code/CPAC/pipeline/engine.py", line 1097, in connect_block
    pipe_x, opt)
  File "/code/CPAC/sca/sca.py", line 505, in dual_regression
    f'spatial_map_dataflow_for_DR_{pipe_num}'
  File "/code/CPAC/utils/datasource.py", line 991, in create_spatial_map_dataflow
    spatial_map_dict[base_name]
ValueError: Files with same name not allowed: s3://fcp-indi/resources/cpac/resources/PNAS_Smith09_rsn10.nii.gz /cpac_templates/PNAS_Smith09_rsn10.nii.gz
shnizzedy commented 3 years ago

regtest-4 with https://github.com/FCP-INDI/C-PAC/pull/1444

Traceback (most recent call last):
  File "/code/run.py", line 677, in <module>
    test_config = 1 if args.analysis_level == "test_config" else 0
  File "/code/CPAC/pipeline/cpac_runner.py", line 549, in run
    p_name, plugin, plugin_args, test_config)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 347, in run_workflow
    subject_id, sub_dict, c, p_name, num_ants_cores
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 1066, in build_workflow
    wf = connect_pipeline(wf, cfg, rpool, pipeline_blocks)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 806, in connect_pipeline
    wf = nb.connect_block(wf, cfg, rpool)
  File "/code/CPAC/pipeline/engine.py", line 1097, in connect_block
    pipe_x, opt)
  File "/code/CPAC/registration/registration.py", line 1676, in register_ANTs_anat_to_template
    params, 'T1w')
  File "/code/CPAC/registration/registration.py", line 1097, in ANTs_registration_connector
    raise Exception(err_msg)
Exception: 

[!] C-PAC says: 
You have selected ANTs as your anatomical registration method.
However, no ANTs parameters were specified.
Please specify ANTs parameters properly and try again.
shnizzedy commented 3 years ago

With the changes in https://github.com/FCP-INDI/C-PAC/pull/1448, I'm now hitting 

Traceback (most recent call last):
  File "/code/run.py", line 677, in <module>
    test_config = 1 if args.analysis_level == "test_config" else 0
  File "/code/CPAC/pipeline/cpac_runner.py", line 549, in run
    p_name, plugin, plugin_args, test_config)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 348, in run_workflow
    subject_id, sub_dict, c, p_name, num_ants_cores
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 1067, in build_workflow
    wf = connect_pipeline(wf, cfg, rpool, pipeline_blocks)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 807, in connect_pipeline
    wf = nb.connect_block(wf, cfg, rpool)
  File "/code/CPAC/pipeline/engine.py", line 1163, in connect_block
    pipe_x, outs)
  File "/code/CPAC/pipeline/engine.py", line 627, in post_process
    node, out = self.get_data(mask, pipe_idx=mask_idx)
  File "/code/CPAC/pipeline/engine.py", line 275, in get_data
    print(self.get(resource)['data'])
KeyError: "When trying to connect node block 'vmhc' to workflow 'cpac_1019436' after node block 'transform_timeseries_to_sym_template': data"

The return value from self.get(resource) looks like 

{"[['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-reproc_bold:func_reorient'], ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'], 'desc-motion_bold:motion_correction_3dvolreg', 'space-bold_desc-brain_mask:bold_mask_afni', 'space-bold_desc-brain_mask:func_normalize']": {
    'data': (cpac_1019436.func_mask_normalize_117, 'out_file'),
    'json': {'CpacProvenance': [
        ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient'],
        ['bold:func_ingress', 'desc-preproc_bold:func_truncate', 'desc-preproc_bold:func_slice_time', 'desc-preproc_bold:func_reorient', 'motion-basefile:get_motion_ref_mean'],
        'desc-motion_bold:motion_correction_3dvolreg',
        'space-bold_desc-brain_mask:bold_mask_afni',
        'space-bold_desc-brain_mask:func_normalize'],
    'Sources': ['desc-brain_bold', 'space-bold_desc-brain_mask']}
}}
shnizzedy commented 3 years ago

If I do https://github.com/FCP-INDI/C-PAC/blob/1ad0f4db6d3ac1d8d10f80c40bf0273cb4f69c80/CPAC/pipeline/engine.py#L618-L619, it gets farther, but raises a new exception:

Connecting smooth_func_vmhc...

Connecting transform_timeseries_to_sym_template...

Connecting vmhc...

Connecting network_centrality...

Traceback (most recent call last):
  File "/code/run.py", line 681, in <module>
    test_config = 1 if args.analysis_level == "test_config" else 0
  File "/code/CPAC/pipeline/cpac_runner.py", line 549, in run
    p_name, plugin, plugin_args, test_config)
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 348, in run_workflow
    subject_id, sub_dict, c, p_name, num_ants_cores
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 1106, in build_workflow
    rpool.gather_pipes(wf, cfg)
  File "/code/CPAC/pipeline/engine.py", line 888, in gather_pipes
    raise Exception("\n[!] No atlas ID found for "
Exception: 
[!] No atlas ID found for 1019436_1_desc-MeanSCA-zstd-1_correlations.
shnizzedy commented 3 years ago

With more changes in https://github.com/FCP-INDI/C-PAC/pull/1448, these warnings are raised, but C-PAC runs:

/code/CPAC/utils/datasource.py:990: UserWarning: File with same name already exists in resource pool.Using existing file s3://fcp-indi/resources/cpac/resources/PNAS_Smith09_rsn10.nii.gz for /cpac_templates/PNAS_Smith09_rsn10.nii.gz
  spatial_map_dict[base_name]
/code/CPAC/pipeline/engine.py:888: UserWarning: 
[!] No atlas ID found for 1019436_1_desc-MeanSCA-zstd-1_correlations.

  LookupError("\n[!] No atlas ID found for "
/code/CPAC/pipeline/engine.py:888: UserWarning: 
[!] No atlas ID found for 1019436_1_desc-MeanSCA-zstd-2_correlations.

  LookupError("\n[!] No atlas ID found for "
/code/CPAC/pipeline/engine.py:888: UserWarning: 
[!] No atlas ID found for 1019436_1_desc-MeanSCA-zstd-3_correlations.

  LookupError("\n[!] No atlas ID found for "
shnizzedy commented 3 years ago

:no_good: (regtest-1)

210223-20:08:13,898 nipype.workflow WARNING:
     Storing result file without outputs
210223-20:08:13,899 nipype.workflow WARNING:
     [Node] Error on "cpac_1019436.anat_skullstrip_40" (/media/ebs/runs/2021-02-23/regtest-1/working/cpac_1019436/anat_skullstrip_40)
210223-20:08:13,902 nipype.utils WARNING:
     No metadata was found in the pkl file. Make sure you are currently using the same Nipype version from the generated pkl.
exception calling callback for <Future at 0x7f7fa498e390 state=finished raised FileNotFoundError>
concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 421, in run
    outputs = self.aggregate_outputs(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 498, in aggregate_outputs
    predicted_outputs = self._list_outputs()  # Predictions from _list_outputs
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/afni/base.py", line 249, in _list_outputs
    outputs = super(AFNICommand, self)._list_outputs()
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 949, in _list_outputs
    outputs[out_name] = os.path.abspath(fname)
  File "/usr/local/miniconda/lib/python3.7/posixpath.py", line 383, in abspath
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 67, in run_node
    result["result"] = node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 471, in run
    vals = np.loadtxt(mon_sp.fname, delimiter=",")
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/npyio.py", line 962, in loadtxt
    fh = np.lib._datasource.open(fname, 'rt', encoding=encoding)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 265, in open
    ds = DataSource(destpath)
  File "/usr/local/miniconda/lib/python3.7/site-packages/numpy/lib/_datasource.py", line 322, in __init__
    self._destpath = os.path.abspath(destpath)
  File "/usr/local/miniconda/lib/python3.7/posixpath.py", line 383, in abspath
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/process.py", line 239, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 70, in run_node
    result["result"] = node.result
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 217, in result
    op.join(self.output_dir(), "result_%s.pklz" % self.name)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 293, in load_resultfile
    result = loadpkl(results_file)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 672, in loadpkl
    raise e
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 648, in loadpkl
    with indirectory(infile.parent):
  File "/usr/local/miniconda/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/utils/filemanip.py", line 949, in indirectory
    cwd = os.getcwd()
FileNotFoundError: [Errno 2] No such file or directory
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 324, in _invoke_callbacks
    callback(self)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/multiproc.py", line 159, in _async_callback
    result = args.result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "/usr/local/miniconda/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
FileNotFoundError: [Errno 2] No such file or directory
shnizzedy commented 3 years ago

After https://github.com/FCP-INDI/C-PAC/pull/1448/commits/74de9d1bbb14b0ccfab0df662ac9eea28d71538c & https://github.com/FCP-INDI/C-PAC/pull/1448/commits/6211d5ee32915d338b0dd10a95ee38ef2082dc8d, runs with just these warnings:

/code/CPAC/pipeline/engine.py:887: UserWarning: 
[!] No atlas ID found for 1019436_1_desc-MeanSCA-zstd-1_correlations.

  LookupError("\n[!] No atlas ID found for "
/code/CPAC/pipeline/engine.py:887: UserWarning: 
[!] No atlas ID found for 1019436_1_desc-MeanSCA-zstd-2_correlations.

  LookupError("\n[!] No atlas ID found for "
/code/CPAC/pipeline/engine.py:887: UserWarning: 
[!] No atlas ID found for 1019436_1_desc-MeanSCA-zstd-3_correlations.

  LookupError("\n[!] No atlas ID found for "