FCP-INDI / cpac_run_logs

Repository for information about C-PAC runs.
1 stars 1 forks source link

app-C-PAC-brainlife.io v0.1.0 / C-PAC v1.8.0 - default pipeline #17

Open shnizzedy opened 3 years ago

shnizzedy commented 3 years ago

C-PAC Run Info

Description TDAfMRI
Version 1.8.0 in C-PAC-brainlife 0.1.0
Container Singularity
System Brainlife.op
Data Size BOLD: 1017 TRs (96 × 96 ×40)
T1W: 256 × 256 × 192
Results CPAC run error

crash-20210408-011807-user-afni_centrality.a0-801f0be8-89c0-4f18-a142-ab8a64e47b5e.txt

tail of pypeline.log (before run summary)

210408-01:18:04,907 nipype.workflow INFO:
     [Node] Running "afni_centrality" ("nipype.interfaces.afni.preprocess.ECM"), a CommandLine Interface with command:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
210408-01:18:06,53 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.052929:++ 3dECM: AFNI version=AFNI_20.0.04 (Mar 15 2021) [64-bit]
210408-01:18:06,175 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
210408-01:18:06,194 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:     Warnings of this type will be muted for this session.
210408-01:18:06,214 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
210408-01:18:06,233 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:double free or corruption (out)
210408-01:18:06,257 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:
210408-01:18:06,275 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:Fatal Signal 6 (SIGABRT) received
210408-01:18:06,301 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:  3dECM main
210408-01:18:06,319 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519: Bottom of Debug Stack
210408-01:18:06,337 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** Command line was:
210408-01:18:06,355 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
210408-01:18:06,374 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** AFNI version = AFNI_20.0.04  Compile date = Mar 15 2021
210408-01:18:06,392 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
210408-01:18:06,410 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** Program Death **
210408-01:18:06,429 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** If you report this crash to the AFNI message board,
210408-01:18:06,446 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** please copy the error messages EXACTLY, and give
210408-01:18:06,463 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** the command line you used to run the program, and
210408-01:18:06,481 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** any other information needed to repeat the problem.
210408-01:18:06,499 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.175519:** You may later be asked to upload data to help debug.
210408-01:18:06,675 nipype.interface INFO:
     stderr 2021-04-08T01:18:06.675929:** Crash log is appended to file /home/user/.afni.crashlog
210408-01:18:07,8 nipype.workflow WARNING:
     Storing result file without outputs
210408-01:18:07,50 nipype.workflow WARNING:
     [Node] Error on "cpac_sub-SBJ01_ses-1.afni_centrality_eigenvector_centrality_274.afni_centrality" (/output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality)
210408-01:18:07,83 nipype.workflow ERROR:
     Node afni_centrality.a0 failed to run on host slurm7-compute9.
210408-01:18:07,103 nipype.workflow ERROR:
     Saving crash info to /output/log/crash-20210408-011807-user-afni_centrality.a0-801f0be8-89c0-4f18-a142-ab8a64e47b5e.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
    node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/afni/base.py", line 125, in _run_interface
    runtime, correct_return_codes
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
Standard output:
Standard error:
++ 3dECM: AFNI version=AFNI_20.0.04 (Mar 15 2021) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
     Warnings of this type will be muted for this session.
     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
double free or corruption (out)
Fatal Signal 6 (SIGABRT) received
  3dECM main
 Bottom of Debug Stack
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI version = AFNI_20.0.04  Compile date = Mar 15 2021
** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
** Program Death **
** If you report this crash to the AFNI message board,
** please copy the error messages EXACTLY, and give
** the command line you used to run the program, and
** any other information needed to repeat the problem.
** You may later be asked to upload data to help debug.
** Crash log is appended to file /home/user/.afni.crashlog
Return code: 1

tail of slurm-18050.err (before run summary)

Traceback (most recent call last):
  File "/code/CPAC/pipeline/cpac_pipeline.py", line 450, in run_workflow
    workflow.run(plugin=plugin, plugin_args=plugin_args)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/workflows.py", line 632, in run
    runner.run(execgraph, updatehash=updatehash, config=self.config)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/linear.py", line 67, in run
    self._status_callback(node, endstatus)
  File "/code/CPAC/utils/monitoring/monitoring.py", line 61, in log_nodes_cb
    runtime = node.result.runtime
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 217, in result
    op.join(self.output_dir(), "result_%s.pklz" % self.name)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/utils.py", line 291, in load_resultfile
    raise FileNotFoundError(results_file)
FileNotFoundError: /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/sep_nifti_subbriks/result_sep_nifti_subbriks.pklz

/output/working/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/command.txt

3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz

/output/working/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/_report/report.rst

Node: afni_centrality_eigenvector_centrality_274 (afni_centrality (afni)

Hierarchy : cpac_sub-SBJ01_ses-1.afni_centrality_eigenvector_centrality_274.afni_centrality Exec ID : afni_centrality.a0

Original Inputs

  • args : \<undefined>
  • autoclip : \<undefined>
  • automask : \<undefined>
  • environ : {'OMP_NUM_THREADS': '1'}
  • eps : \<undefined>
  • fecm : \<undefined>
  • full : \<undefined>
  • in_file : /output/working/cpac_sub-SBJ01_ses-1/_scan_rest/resample_functional_to_template_274/bandpassed_demeaned_filtered_trans_flirt.nii.gz
  • mask : /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz
  • max_iter : \<undefined>
  • memory : 1.0
  • num_threads : 1
  • out_file : eigenvector_centrality_merged.nii.gz
  • outputtype : AFNI
  • polort : \<undefined>
  • scale : \<undefined>
  • shift : \<undefined>
  • sparsity : 0.001
  • thresh : \<undefined>

Include:

Developers only:

shnizzedy commented 3 years ago

Running 3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz outside of C-PAC / brainlife replicates. Here's the AFNI crash log:

*********------ CRASH LOG ------------------------------***********
Fatal Signal 11 (SIGSEGV) received
.......... recent internal history .........................................
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {49 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {49 ms}
     mri_free -- call killpurge {49 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {49 ms}
      mri_killpurge -- check if im==NULL ptr=0x562e5d6d6d80
      mri_killpurge -- can't killpurge NULL fname! {49 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {49 ms}
     mri_free -- free im {49 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {49 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {49 ms}
     mri_free -- call killpurge {49 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {49 ms}
      mri_killpurge -- check if im==NULL ptr=0x562e5d6d6e40
      mri_killpurge -- can't killpurge NULL fname! {49 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {49 ms}
     mri_free -- free im {49 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {49 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {49 ms}
     mri_free -- call killpurge {49 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {49 ms}
      mri_killpurge -- check if im==NULL ptr=0x562e5d6d6f00
      mri_killpurge -- can't killpurge NULL fname! {49 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {49 ms}
     mri_free -- free im {49 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {49 ms}
    THD_delete_datablock -- free brick_ stuff {49 ms}
    THD_delete_datablock -- KILL_KILL {49 ms}
    THD_delete_datablock -- free attributes {49 ms}
----THD_delete_datablock [4]: EXIT} (file=thd_delete.c line=122) to THD_delete_3dim_dataset {49 ms}
   THD_delete_3dim_dataset -- KILL_KILL {49 ms}
---THD_delete_3dim_dataset [3]: EXIT} (file=thd_delete.c line=179) to 3dECM main {49 ms}
............................................................................
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz
............................................................................
  3dECM main
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI compile date = Mar 15 2021
** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
** Program Crash **
shnizzedy commented 3 years ago

If I increase the memory to 2 (3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 2.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz)

++ 3dECM: AFNI version=AFNI_20.0.04 (Mar 15 2021) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
++ 49428 voxels in -mask dataset
++ vectim-izing input dataset
++ Detrending with polort = 1
++ Calculating ECM with full method (sparsity=0.001%, thresh=-1.200, scale=1.000, shift=0.000,
max_iter=10000, eps=0.000, binary=0, mem=1946212768)
++ Starting create_sparse_corr_array with a memory allowance of 1946212768
++ The sparse array with 12216 values will take 0.37 MB of memory (budget = 1855.67 MB)
++ Extracting sparse correlation array with threshold = -1.200000 and sparsity = 0.00% (12216)
++ 8 OpenMP threads started
Looping:0Killed
shnizzedy commented 3 years ago

bandpassed_demeaned_filtered_trans_flirt.nii.gz doesn't appear to be incomplete or corrupted: fsleyes screenshot of `bandpassed_demeaned_filtered_trans_flirt.nii.gz`

shnizzedy commented 3 years ago

Trying -thresh instead of -sparsity (3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -thresh 0.999 bandpassed_demeaned_filtered_trans_flirt.nii.gz) yields

*********------ CRASH LOG ------------------------------***********
Fatal Signal 11 (SIGSEGV) received
.......... recent internal history .........................................
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {32 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {32 ms}
     mri_free -- call killpurge {32 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {32 ms}
      mri_killpurge -- check if im==NULL ptr=0x55f33ae02b00
      mri_killpurge -- can't killpurge NULL fname! {32 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {32 ms}
     mri_free -- free im {32 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {32 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {32 ms}
     mri_free -- call killpurge {32 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {32 ms}
      mri_killpurge -- check if im==NULL ptr=0x55f33adca580
      mri_killpurge -- can't killpurge NULL fname! {32 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {32 ms}
     mri_free -- free im {32 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {32 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {32 ms}
     mri_free -- call killpurge {32 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {32 ms}
      mri_killpurge -- check if im==NULL ptr=0x55f33adca640
      mri_killpurge -- can't killpurge NULL fname! {32 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {32 ms}
     mri_free -- free im {32 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {32 ms}
    THD_delete_datablock -- free brick_ stuff {32 ms}
    THD_delete_datablock -- KILL_KILL {32 ms}
    THD_delete_datablock -- free attributes {32 ms}
----THD_delete_datablock [4]: EXIT} (file=thd_delete.c line=122) to THD_delete_3dim_dataset {32 ms}
   THD_delete_3dim_dataset -- KILL_KILL {32 ms}
---THD_delete_3dim_dataset [3]: EXIT} (file=thd_delete.c line=179) to 3dECM main {32 ms}
............................................................................
  3dECM main
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -thresh 0.999 bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI compile date = Mar 15 2021
** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
** Program Crash **

Trying -automask instead of -mask (3dECM -automask -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz) yields

*********------ CRASH LOG ------------------------------***********
Fatal Signal 11 (SIGSEGV) received
.......... recent internal history .........................................
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {31 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {31 ms}
     mri_free -- call killpurge {31 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {31 ms}
      mri_killpurge -- check if im==NULL ptr=0x5654cee58b90
      mri_killpurge -- can't killpurge NULL fname! {31 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {31 ms}
     mri_free -- free im {31 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {31 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {31 ms}
     mri_free -- call killpurge {31 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {31 ms}
      mri_killpurge -- check if im==NULL ptr=0x5654cee58c50
      mri_killpurge -- can't killpurge NULL fname! {31 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {31 ms}
     mri_free -- free im {31 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {31 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {31 ms}
     mri_free -- call killpurge {31 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {31 ms}
      mri_killpurge -- check if im==NULL ptr=0x5654cee58d10
      mri_killpurge -- can't killpurge NULL fname! {31 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {31 ms}
     mri_free -- free im {31 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {31 ms}
    THD_delete_datablock -- free brick_ stuff {31 ms}
    THD_delete_datablock -- KILL_KILL {31 ms}
    THD_delete_datablock -- free attributes {31 ms}
----THD_delete_datablock [4]: EXIT} (file=thd_delete.c line=122) to THD_delete_3dim_dataset {31 ms}
   THD_delete_3dim_dataset -- KILL_KILL {31 ms}
---THD_delete_3dim_dataset [3]: EXIT} (file=thd_delete.c line=179) to 3dECM main {31 ms}
............................................................................
  3dECM main
** Command line was:
3dECM -automask -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI compile date = Mar 15 2021
** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
** Program Crash **
sgiavasis commented 3 years ago

To report here - I ran this with the same data, but on an AFNI version of 3dECM with compile date Jan 3 2017.

I see above that yours got killed right after the Looping: bit shows up. Mine gets a little farther, and then crashes with a bit more information to terminal:

3dECM -mask Mask_ABIDE_85Percent_GM.nii.gz -memory 2.000000 -prefix eigencent.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz

++ 3dECM: AFNI version=AFNI_17.0.00 (Jan 3 2017) [64-bit] AFNI converts NIFTI_datatype=64 (FLOAT64) in file bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32 Warnings of this type will be muted for this session. Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none. ++ 49428 voxels in -mask dataset ++ vectim-izing input dataset ++ Detrending with polort = 1 ++ Calculating ECM with full method (sparsity=0.001%, thresh=-1.200, scale=1.000, shift=0.000, max_iter=10000, eps=0.000, binary=0, mem=1946212768) ++ Starting create_sparse_corr_array with a memory allowance of 1946212768 ++ The sparse array with 12216 values will take 0.37 MB of memory (budget = 1855.67 MB) ++ Extracting sparse correlation array with threshold = -1.200000 and sparsity = 0.00% (12216) ++ 8 OpenMP threads started Looping:0+ WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! 123456+ WARNING: Could not allocate a new node! 789,0123456789,0123456789,0123456789,0123456789,. ERROR: Memory budget (1856.053131 MB) exceeded, consider using ahigher correlation or lower sparsity threshold + WARNING: Error getting sparse weight array. ++ Calculating ECM with full method (sparsity=0.001%, thresh=-1.200, scale=1.000, shift=0.000, max_iter=10000, eps=0.000, binary=1, mem=1945421920) ++ Starting create_sparse_corr_array with a memory allowance of 1945421920 ++ The sparse array with 12216 values will take 0.37 MB of memory (budget = 1854.92 MB) ++ Extracting sparse correlation array with threshold = -1.200000 and sparsity = 0.00% (12216) ++ 8 OpenMP threads started Looping:0+ WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! + WARNING: Could not allocate a new node! 1234+ WARNING: Could not allocate a new node! 56789,01234+ WARNING: Could not allocate a new node! 56789,0+ WARNING: Could not allocate a new node! 12345*+ WARNING: Could not allocate a new node! 6789,0123456789,0123456789,. * ERROR: Memory budget (1855.298920 MB) exceeded, consider using ahigher correlation or lower sparsity threshold + WARNING: Error getting sparse weight array. creating output bucket(eigencent.nii.gz)

  • creating output dataset in memory

Fatal Signal 11 (SIGSEGV) received 3dECM main Bottom of Debug Stack Command line was: 3dECM -mask Mask_ABIDE_85Percent_GM.nii.gz -memory 2.000000 -prefix eigencent.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz AFNI version = AFNI_17.0.00 Compile date = Jan 3 2017 [[Precompiled binary macosx_10.7_Intel_64: Jan 3 2017]] Program Death If you report this crash to the AFNI message board, please copy the error messages EXACTLY, and give the command line you used to run the program, and any other information needed to repeat the problem. You may later be asked to upload data to help debug. ** Crash log is appended to file /Users/steven.giavasis/.afni.crashlog

Here's the crash log:

*------ CRASH LOG ------------------------------*** Fatal Signal 11 (SIGSEGV) received .......... recent internal history ......................................... +++THD_find_atr [3]: {ENTRY (file=thd_atr.c line=408) from 3dECM main ---THD_find_atr [3]: EXIT} (file=thd_atr.c line=413) to 3dECM main +++AFNI_process_environ [3]: {ENTRY (file=afni_environ.c line=112) from 3dECM main ++++AFNI_suck_file [4]: {ENTRY (file=afni_environ.c line=25) from AFNI_process_environ ----AFNI_suck_file [4]: EXIT} (file=afni_environ.c line=30) to AFNI_process_environ ---AFNI_process_environ [3]: EXIT} (file=afni_environ.c line=132) to 3dECM main +++THD_set_char_atr [3]: {ENTRY (file=thd_atr.c line=651) from 3dECM main ++++THD_set_atr [4]: {ENTRY (file=thd_atr.c line=498) from THD_set_char_atr THD_set_atr -- HISTORY_NOTE +++++THD_find_atr [5]: {ENTRY (file=thd_atr.c line=408) from THD_set_atr -----THD_find_atr [5]: EXIT} (file=thd_atr.c line=413) to THD_set_atr ----THD_set_atr [4]: EXIT} (file=thd_atr.c line=623) to THD_set_char_atr ---THD_set_char_atr [3]: EXIT} (file=thd_atr.c line=653) to 3dECM main +++EDIT_dset_items [3]: {ENTRY (file=edt_dsetitems.c line=125) from 3dECM main ---EDIT_dset_items [3]: EXIT} (file=edt_dsetitems.c line=964) to 3dECM main +++EDIT_dset_items [3]: {ENTRY (file=edt_dsetitems.c line=125) from 3dECM main ---EDIT_dset_items [3]: EXIT} (file=edt_dsetitems.c line=964) to 3dECM main +++EDIT_dset_items [3]: {ENTRY (file=edt_dsetitems.c line=125) from 3dECM main ---EDIT_dset_items [3]: EXIT} (file=edt_dsetitems.c line=964) to 3dECM main +++EDIT_substitute_brick [3]: {ENTRY (file=edt_substbrick.c line=33) from 3dECM main ++++mri_new_7D_generic [4]: {ENTRY (file=mri_new.c line=48) from EDIT_substitute_brick ----mri_new_7D_generic [4]: EXIT} (file=mri_new.c line=142) to EDIT_substitute_brick ++++mri_free [4]: {ENTRY (file=mri_free.c line=49) from EDIT_substitute_brick mri_free -- call killpurge +++++mri_killpurge [5]: {ENTRY (file=mri_purger.c line=259) from mri_free mri_killpurge -- check if im==NULL ptr=0x7f9b68300590 mri_killpurge -- can't killpurge NULL fname! -----mri_killpurge [5]: EXIT} (file=mri_purger.c line=270) to mri_free mri_free -- free im ----mri_free [4]: EXIT} (file=mri_free.c line=67) to EDIT_substitute_brick ---EDIT_substitute_brick [3]: EXIT} (file=edt_substbrick.c line=67) to 3dECM main ............................................................................ Command line was: 3dECM -mask Mask_ABIDE_85Percent_GM.nii.gz -memory 2.000000 -prefix eigencent.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz ............................................................................ 3dECM main Command line was: 3dECM -mask Mask_ABIDE_85Percent_GM.nii.gz -memory 2.000000 -prefix eigencent.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz AFNI compile date = Jan 3 2017 [[Precompiled binary macosx_10.7_Intel_64: Jan 3 2017]] Program Crash

Note:

* ERROR: Memory budget (1855.298920 MB) exceeded, consider using ahigher correlation or lower sparsity threshold + WARNING: Error getting sparse weight array. creating output bucket(eigencent.nii.gz)

creating output dataset in memory Fatal Signal 11 (SIGSEGV) received

shnizzedy commented 3 years ago

Just confirmed, same behavior in Singularity on AWS.

210415-21:25:48,370 nipype.workflow WARNING:
     [Node] Error on "cpac_sub-SBJ01_ses-1.afni_centrality_eigenvector_centrality_274.afni_centrality" (/media/ebs/brainlife/out_2021-04-13/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality)
210415-21:25:48,374 nipype.workflow ERROR:
     Node afni_centrality.a0 failed to run on host ip-172-31-79-78.
210415-21:25:48,440 nipype.workflow ERROR:
     Saving crash info to /media/ebs/brainlife/out_2021-04-13/log/crash-20210415-212548-ubuntu-afni_centrality.a0-d542083b-5a2d-426d-a8bf-d1f1ed04f7b2.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
    node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/afni/base.py", line 125, in _run_interface
    runtime, correct_return_codes
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /media/ebs/brainlife/out_2021-04-13/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
Standard output:

Standard error:
++ 3dECM: AFNI version=AFNI_20.0.04 (Mar 15 2021) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /media/ebs/brainlife/out_2021-04-13/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
     Warnings of this type will be muted for this session.
     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.

Fatal Signal 11 (SIGSEGV) received
  3dECM main
 Bottom of Debug Stack
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /media/ebs/brainlife/out_2021-04-13/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI version = AFNI_20.0.04  Compile date = Mar 15 2021
** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
** Program Death **
** If you report this crash to the AFNI message board,
** please copy the error messages EXACTLY, and give
** the command line you used to run the program, and
** any other information needed to repeat the problem.
** You may later be asked to upload data to help debug.
** Crash log is appended to file /home/ubuntu/.afni.crashlog
Return code: 1

Next I will try in Docker on AWS.

shnizzedy commented 3 years ago

Same result in Docker:

210416-04:46:16,949 nipype.workflow INFO:
         [Node] Setting-up "cpac_sub-SBJ01_ses-1.afni_centrality_eigenvector_centrality_274.afni_centrality" in "/media/ebs/brainlife/out_2021-04-15/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvect
or_centrality_274/_scan_rest/afni_centrality".
210416-04:46:16,955 nipype.workflow INFO:
         [Node] Running "afni_centrality" ("nipype.interfaces.afni.preprocess.ECM"), a CommandLine Interface with command:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /media/ebs/brainlife/out_2021-04-15/working/cpac_sub-SBJ01_ses-
1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
210416-04:46:17,251 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.251403:++ 3dECM: AFNI version=AFNI_20.0.04 (Mar 15 2021) [64-bit]
210416-04:46:17,277 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.277298:** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /media/ebs/brainlife/out_2021-04-15/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274
/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
210416-04:46:17,277 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.277298:     Warnings of this type will be muted for this session.
210416-04:46:17,278 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.277298:     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
210416-04:46:17,311 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:
210416-04:46:17,311 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:Fatal Signal 11 (SIGSEGV) received
210416-04:46:17,312 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:  3dECM main
210416-04:46:17,312 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404: Bottom of Debug Stack
210416-04:46:17,312 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** Command line was:
210416-04:46:17,312 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /media/ebs/brainlife
/out_2021-04-15/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
210416-04:46:17,313 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** AFNI version = AFNI_20.0.04  Compile date = Mar 15 2021
210416-04:46:17,313 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
210416-04:46:17,313 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** Program Death **
210416-04:46:17,313 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** If you report this crash to the AFNI message board,
210416-04:46:17,314 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** please copy the error messages EXACTLY, and give
210416-04:46:17,314 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** the command line you used to run the program, and
210416-04:46:17,314 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** any other information needed to repeat the problem.
210416-04:46:17,314 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.311404:** You may later be asked to upload data to help debug.
210416-04:46:17,315 nipype.interface INFO:
         stderr 2021-04-16T04:46:17.315104:** Crash log is appended to file /root/.afni.crashlog
210416-04:46:17,547 nipype.workflow WARNING:
         Storing result file without outputs
210416-04:46:17,548 nipype.workflow WARNING:
         [Node] Error on "cpac_sub-SBJ01_ses-1.afni_centrality_eigenvector_centrality_274.afni_centrality" (/media/ebs/brainlife/out_2021-04-15/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality)
210416-04:46:17,553 nipype.workflow ERROR:
         Node afni_centrality.a0 failed to run on host d1708d22d5c5.
210416-04:46:17,555 nipype.workflow ERROR:
         Saving crash info to /media/ebs/brainlife/out_2021-04-15/log/crash-20210416-044617-root-afni_centrality.a0-683396c6-1455-41d4-9da4-cd3d363910fd.txt
Traceback (most recent call last):
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/plugins/linear.py", line 46, in run
    node.run(updatehash=updatehash)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 516, in run
    result = self._run_interface(execute=True)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 635, in _run_interface
    return self._run_command(execute)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/pipeline/engine/nodes.py", line 741, in _run_command
    result = self._interface.run(cwd=outdir)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 419, in run
    runtime = self._run_interface(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/afni/base.py", line 125, in _run_interface
    runtime, correct_return_codes
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 814, in _run_interface
    self.raise_exception(runtime)
  File "/usr/local/miniconda/lib/python3.7/site-packages/nipype/interfaces/base/core.py", line 745, in raise_exception
    ).format(**runtime.dictcopy())
RuntimeError: Command:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /media/ebs/brainlife/out_2021-04-15/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
Standard output:

Standard error:
++ 3dECM: AFNI version=AFNI_20.0.04 (Mar 15 2021) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /media/ebs/brainlife/out_2021-04-15/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
     Warnings of this type will be muted for this session.
     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.

Fatal Signal 11 (SIGSEGV) received
  3dECM main
 Bottom of Debug Stack
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 /media/ebs/brainlife/out_2021-04-15/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI version = AFNI_20.0.04  Compile date = Mar 15 2021
** [[Precompiled binary linux_ubuntu_16_64: Mar 15 2021]]
** Program Death **
** If you report this crash to the AFNI message board,
** please copy the error messages EXACTLY, and give
** the command line you used to run the program, and
** any other information needed to repeat the problem.
** You may later be asked to upload data to help debug.
** Crash log is appended to file /root/.afni.crashlog
Return code: 1
shnizzedy commented 3 years ago

Also with a newer version of AFNI:

++ 3dECM: AFNI version=AFNI_21.0.18 (Mar 14 2021) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /Downloads/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
     Warnings of this type will be muted for this session.
     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
munmap_chunk(): invalid pointer

Fatal Signal 6 (SIGABRT) received
  3dECM main
 Bottom of Debug Stack
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI version = AFNI_21.0.18  Compile date = Mar 14 2021
** [[Precompiled binary linux_openmp_64: Mar 14 2021]]
** Program Death **
** If you report this crash to the AFNI message board,
** please copy the error messages EXACTLY, and give
** the command line you used to run the program, and
** any other information needed to repeat the problem.
** You may later be asked to upload data to help debug.
** Crash log is appended to file /root/.afni.crashlog
*********------ CRASH LOG ------------------------------***********
Fatal Signal 6 (SIGABRT) received
.......... recent internal history .........................................
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {30 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {30 ms}
     mri_free -- call killpurge {30 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {30 ms}
      mri_killpurge -- check if im==NULL ptr=0x24277f0
      mri_killpurge -- can't killpurge NULL fname! {30 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {30 ms}
     mri_free -- free im {30 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {30 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {30 ms}
     mri_free -- call killpurge {30 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {30 ms}
      mri_killpurge -- check if im==NULL ptr=0x24278b0
      mri_killpurge -- can't killpurge NULL fname! {30 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {30 ms}
     mri_free -- free im {30 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {30 ms}
+++++mri_free [5]: {ENTRY (file=mri_free.c line=49) from THD_delete_datablock {30 ms}
     mri_free -- call killpurge {30 ms}
++++++mri_killpurge [6]: {ENTRY (file=mri_purger.c line=259) from mri_free {30 ms}
      mri_killpurge -- check if im==NULL ptr=0x2427970
      mri_killpurge -- can't killpurge NULL fname! {30 ms}
------mri_killpurge [6]: EXIT} (file=mri_purger.c line=270) to mri_free {30 ms}
     mri_free -- free im {30 ms}
-----mri_free [5]: EXIT} (file=mri_free.c line=67) to THD_delete_datablock {30 ms}
    THD_delete_datablock -- free brick_ stuff {30 ms}
    THD_delete_datablock -- KILL_KILL {30 ms}
    THD_delete_datablock -- free attributes {30 ms}
----THD_delete_datablock [4]: EXIT} (file=thd_delete.c line=122) to THD_delete_3dim_dataset {30 ms}
   THD_delete_3dim_dataset -- KILL_KILL {30 ms}
---THD_delete_3dim_dataset [3]: EXIT} (file=thd_delete.c line=179) to 3dECM main {30 ms}
............................................................................
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz
............................................................................
  3dECM main
** Command line was:
3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 1.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.001000 bandpassed_demeaned_filtered_trans_flirt.nii.gz
** AFNI compile date = Mar 14 2021
** [[Precompiled binary linux_openmp_64: Mar 14 2021]]
** Program Crash **
shnizzedy commented 3 years ago

I made some fixes to 3dECM that were pushed to afni about 10 days ago, so they are not in the build that you are testing. I tested with the newer version and was not able to reproduce the problem.

I think that the problem is arising from the amount of memory you are allocating for the operation. You are specifying 1 GB, but the compressed input data alone is almost 500MB. This is triggering an error message and then frees all of the memory to exit. There is an errant double free in this process that is giving you the exception

there is something particular about this dataset that doesn't work well with the adaptive thresholding procedure that I use in 3dECM. If you give it 5 GB of RAM it completes and says that its peak memory requirement was 2300 MB, but I am having trouble getting it to complete with less than 5GB allocated, there must be a mistake in the memory calculations, that I will try to track down

BTW, the new 3dECM will only calculate weighted or binary centrality, not both as before, so CPAC will need to be slightly modified to account for that. ― @ccraddock on brainhack Mattermost, Saturday, Apr 17, 2021 11:38 AM

shnizzedy commented 3 years ago

I was able to make some progress on this. The dataset that you are working with as a large number of in-mask voxels that have a zero-variance time course. This creates are hug number of correlations == 0, which mucks up the histogram method that I developed for the auto-threshold learning procedure that is used to speedup and resources memory for sparsity.

I added a variance filter and now it works fine on the that dataset. I am planning to look over the changes again before pushing to AFNI, but these fixes should be available soon. ― @ccraddock on brainhack Mattermost, today, 1:46 PM

shnizzedy commented 3 years ago

With the newer 3dECM, I still get a crash (

3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 5.000000 -prefix eigenvector_centrality_merged.nii.gz -sparsity 0.100000 bandpassed_demeaned_filtered_trans_flirt.nii.gz
Standard output:
opened input dataset /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz
Standard error:
++ 3dECM cc mods: AFNI version=AFNI_21.1.02 (Apr 16 2021) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /output/working/cpac_sub-SBJ01_ses-1/afni_centrality_eigenvector_centrality_274/_scan_rest/afni_centrality/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
     Warnings of this type will be muted for this session.
     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
++ 49428 voxels in -mask dataset
++ vectim-izing input dataset
++ Detrending with polort = 1
++ Calculating ECM with full method (sparsity=0.100%,
 thresh=-1.200, scale=1.000, shift=0.000, max_iter=10000, eps=0.000,
 binary=0, mem=5167438240)
++ Starting create_sparse_corr_array with a memory allowance of 5167438240
++ The sparse array with 1221539 values will take 37.28 MB of memory (budget = 4927.67 MB)
++ Extracting sparse correlation array with threshold = -1.200000 and sparsity = 0.10% (1221539)
++ 1 OpenMP threads started
Looping:012*+ WARNING: Could not allocate a new node!
3456789,0123456789,0123456789,0123456789,0123456789,.
** ERROR: Memory budget (4928.053131 MB) exceeded, consider using a higher correlation or lower sparsity threshold
*+ WARNING: Error getting sparse weight array.
** FATAL ERROR: Eigen vector calculation failed!
** Program compile date = Apr 16 2021

) if I leave the default threshold (-1.2) or if I set -thresh (e.g., to 0.99) and leave sparsity to its default (100.00%) (

3dECM -mask /cpac_templates/Mask_ABIDE_85Percent_GM.nii.gz -memory 5.000000 -prefix eigenvector_centrality_merged.nii.gz -thresh 0.999 bandpassed_demeaned_filtered_trans_flirt.nii.gz 
++ 3dECM cc mods: AFNI version=AFNI_21.1.02 (Apr 16 2021) [64-bit]
** AFNI converts NIFTI_datatype=64 (FLOAT64) in file /Downloads/bandpassed_demeaned_filtered_trans_flirt.nii.gz to FLOAT32
     Warnings of this type will be muted for this session.
     Set AFNI_NIFTI_TYPE_WARN to YES to see them all, NO to see none.
opened input dataset bandpassed_demeaned_filtered_trans_flirt.nii.gz
++ 49428 voxels in -mask dataset
++ vectim-izing input dataset
++ Detrending with polort = 1
++ Calculating ECM with full method (sparsity=100.000%,
 thresh=0.999, scale=1.000, shift=0.000, max_iter=10000, eps=0.000,
 binary=0, mem=5167438240)
++ Starting create_sparse_corr_array with a memory allowance of 5167438240
*+ WARNING: Cannot pre-calculate the memory required for a sparse matrix when only a correlation threshold is used. Instead the mem is tracked and if we exceed what would be used by the non-sparse array, the operation will be aborted.
++ Extracting sparse correlation array with threshold = 0.999000 and sparsity = 100.00% (0)
++ 8 OpenMP threads started
Looping:0123456789,0123456789,0123456789,0123456789,0123456789,.
** ERROR: No correlations exceeded threshold, consider using a lower correlation threshold
*+ WARNING: Error getting sparse weight array.
** FATAL ERROR: Eigen vector calculation failed!
** Program compile date = Apr 16 2021

).

However, if I set -thresh 0.99 -sparsity 0.1, 3dECM succeeds but does not converge:

++ Finished iter 10000: Verr 0.000, Vnorm 1.000
*+ WARNING: Power iteration did not converge (0.000 >= 0.000)
in 10000 iterations. You might consider increase max_iters, or
epsilon. For now we are writing out the obtained solution,
which may or may not be good enough.
creating output bucket(eigenvector_centrality_merged.nii.gz)
 + creating output dataset in memory
++ Done..
++ Writing output dataset to disk [1,086,532 bytes]
++ Output dataset ./eigenvector_centrality_merged.nii.gz

.

All this :point_up_2: is pre-addition of the variance filter.