BF: Fix sample aggregation

[asmacdo]

This began as work to add stat, which as a dumb initial implementation was just a list of all the values we get. (to be replaced by a counter). However, the sample aggregation returned strange results-- the averages reported the correct number of samples, but stat was only showing a list of length 1.

So this PR refactors the complexity of sample aggregation be abstracted by a Report function update_from_sample. This will be much more testable.

DONE test update_from_sample~

DONE: since aggregate (was max) is now updating the averages, we can drop Report.averages, since that information is duplicated by report.max_values.averages, which should be renamed to report.full_execution_stats (I don't like that name) or similar.

At the end of the run:

ipdb> report.averages
Averages(rss=1802240.0, vsz=226902016.0, pmem=0.0, pcpu=0.0, num_samples=18)
ipdb> report.max_values.averages
Averages(rss=1802240.0, vsz=226902016.0, pmem=0.0, pcpu=0.0, num_samples=18)
ipdb> 

I've removed the stat update, to add separately in another PR

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.97%. Comparing base (2b12679) to head (08dcaeb). Report is 24 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #180 +/- ## ========================================== + Coverage 94.92% 94.97% +0.05% ========================================== Files 2 2 Lines 473 478 +5 Branches 73 73 ========================================== + Hits 449 454 +5 Misses 12 12 Partials 12 12 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[asmacdo]

Sanity Check

On typhon with singularity (with and without containers run), prior to this PR we got divergent results that clearly showed that duct sample aggregation was not working as expected.

I've rerun those experiments, and now the 2 ways of using singularity with duct are returning very similar outputs.

Singularity run direct

  "execution_summary": {
    "exit_code": 0,
    "command": "singularity run --contain --bind /home/asmacdo/devel/sandbox/mriqc-sanity/sourcedata:/data:ro --bind /home/asmacdo/devel/sandbox/mriqc-sanity/dfix-sr-1:/out --bind /home/asmacdo/devel/sandbox/mriqc-sanity/dfix-sr-1/workdir:/workdir code/containers/images/bids/bids-mriqc--0.16.0.sing /data /out participant --participant-label 02 -w /workdir --no-sub",
    "logs_prefix": "dfix-sr-1/duct_",
    "wall_clock_time": 381.45661306381226,
    "peak_rss": 8019558400,
    "average_rss": 6542514699.13043,
    "peak_vsz": 23663247360,
    "average_vsz": 20365962395.826088,
    "peak_pmem": 0,
    "average_pmem": 0,
    "peak_pcpu": 3438.4,
    "average_pcpu": 332.87663043478267,
    "num_samples": 368,
    "num_reports": 8
  },

Containers Run

  "execution_summary": {
    "exit_code": 0,
    "command": "./code/containers/scripts/singularity_cmd run code/containers/images/bids/bids-mriqc--0.16.0.sing sourcedata dfix-cr-1 participant --participant-label 02 -w dfix-cr-1/workdir --no-sub",
    "logs_prefix": "dfix-cr-1/duct_",
    "wall_clock_time": 377.5985689163208,
    "peak_rss": 8124411904,
    "average_rss": 6525882558.772606,
    "peak_vsz": 23824408576,
    "average_vsz": 20357140132.120544,
    "peak_pmem": 0,
    "average_pmem": 0,
    "peak_pcpu": 3035.2,
    "average_pcpu": 335.2660273972601,
    "num_samples": 365,
    "num_reports": 8
  },

[yarikoptic]

somewhat unrelated I guess: "peak_pcpu": 3438.4, looks odd, even for typhon with 32 cores. Having a plot would have been useful to grasp situation on that

[asmacdo]

Using the poc https://github.com/asmacdo/fancy-duct (2 lines had to change, fixed upstream)

Singularity Direct

Containers Run

[asmacdo]

The case I can think of that could artificially inflate the peak would be if the processes were trading resources back and forth, but the aggregation was totaling the peaks of both. So I added a test for that case, which passes.

IIUC (still could be wrong) this means that the 3438% spike was actually recorded by a single call to ps and the individual processes actually somehow added up to more than numcores*100.

FWIW the record of the spike:

```json { "timestamp": "2024-09-19T14:24:03.175972-04:00", "num_samples": 58, "processes": { "4159232": { "pcpu": 100.0, "pmem": 0.0, "rss": 218427392, "vsz": 705040384, "timestamp": "2024-09-19T14:24:03.175328-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159233": { "pcpu": 104.0, "pmem": 0.0, "rss": 119971840, "vsz": 365043712, "timestamp": "2024-09-19T14:24:03.175350-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159234": { "pcpu": 104.0, "pmem": 0.0, "rss": 119971840, "vsz": 365051904, "timestamp": "2024-09-19T14:24:03.175372-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159235": { "pcpu": 109.0, "pmem": 0.0, "rss": 119734272, "vsz": 365043712, "timestamp": "2024-09-19T14:24:03.175394-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159236": { "pcpu": 104.0, "pmem": 0.0, "rss": 260890624, "vsz": 745840640, "timestamp": "2024-09-19T14:24:03.175414-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159237": { "pcpu": 104.0, "pmem": 0.0, "rss": 119451648, "vsz": 365043712, "timestamp": "2024-09-19T14:24:03.175436-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159238": { "pcpu": 109.0, "pmem": 0.0, "rss": 119971840, "vsz": 365043712, "timestamp": "2024-09-19T14:24:03.175459-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159239": { "pcpu": 104.0, "pmem": 0.0, "rss": 119767040, "vsz": 365043712, "timestamp": "2024-09-19T14:24:03.175481-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159240": { "pcpu": 115.0, "pmem": 0.0, "rss": 119902208, "vsz": 365035520, "timestamp": "2024-09-19T14:24:03.175502-04:00", "etime": "00:49", "cmd": "S /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159241": { "pcpu": 105.0, "pmem": 0.0, "rss": 305442816, "vsz": 791502848, "timestamp": "2024-09-19T14:24:03.175525-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159242": { "pcpu": 110.0, "pmem": 0.0, "rss": 119947264, "vsz": 365031424, "timestamp": "2024-09-19T14:24:03.175549-04:00", "etime": "00:49", "cmd": "S /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159243": { "pcpu": 110.0, "pmem": 0.0, "rss": 217567232, "vsz": 702205952, "timestamp": "2024-09-19T14:24:03.175571-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159244": { "pcpu": 110.0, "pmem": 0.0, "rss": 119894016, "vsz": 365031424, "timestamp": "2024-09-19T14:24:03.175592-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159245": { "pcpu": 105.0, "pmem": 0.0, "rss": 119902208, "vsz": 365035520, "timestamp": "2024-09-19T14:24:03.175613-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159246": { "pcpu": 115.0, "pmem": 0.0, "rss": 119853056, "vsz": 365035520, "timestamp": "2024-09-19T14:24:03.175634-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159247": { "pcpu": 110.0, "pmem": 0.0, "rss": 119955456, "vsz": 365031424, "timestamp": "2024-09-19T14:24:03.175655-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159248": { "pcpu": 115.0, "pmem": 0.0, "rss": 119803904, "vsz": 365031424, "timestamp": "2024-09-19T14:24:03.175675-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159249": { "pcpu": 110.0, "pmem": 0.0, "rss": 119943168, "vsz": 365031424, "timestamp": "2024-09-19T14:24:03.175696-04:00", "etime": "00:49", "cmd": "S /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159250": { "pcpu": 110.0, "pmem": 0.0, "rss": 119955456, "vsz": 365035520, "timestamp": "2024-09-19T14:24:03.175717-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159251": { "pcpu": 105.0, "pmem": 0.0, "rss": 119787520, "vsz": 365031424, "timestamp": "2024-09-19T14:24:03.175737-04:00", "etime": "00:49", "cmd": "S /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159757": { "pcpu": 2.3, "pmem": 0.0, "rss": 10940416, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175886-04:00", "etime": "00:41", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(8)" }, "4159756": { "pcpu": 3.5, "pmem": 0.0, "rss": 10854400, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175865-04:00", "etime": "00:41", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(8)" }, "4158843": { "pcpu": 4.2, "pmem": 0.0, "rss": 14962688, "vsz": 31133696, "timestamp": "2024-09-19T14:24:03.175047-04:00", "etime": "00:57", "cmd": "S /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159768": { "pcpu": 0.0, "pmem": 0.0, "rss": 716800, "vsz": 4620288, "timestamp": "2024-09-19T14:23:33.076612-04:00", "etime": "00:10", "cmd": "S /bin/sh -c 3dvolreg -Fourier -twopass -1Dfile slice.1D -1Dmatrix_save slice.aff12.1D -prefix slice_volreg.nii.gz -zpad 4 -maxdisp1D slice_md.1D /workdir/mriqc_wf/funcMRIQC/fMRI_HMC/_in_file_..data..sub-02..func..sub-02_task-rhymejudgment_bold.nii.gz/gen_ref/slice.nii.gz" }, "4159769": { "pcpu": 103.0, "pmem": 0.0, "rss": 18223104, "vsz": 59785216, "timestamp": "2024-09-19T14:23:33.076633-04:00", "etime": "00:10", "cmd": "R 3dvolreg -Fourier -twopass -1Dfile slice.1D -1Dmatrix_save slice.aff12.1D -prefix slice_volreg.nii.gz -zpad 4 -maxdisp1D slice_md.1D /workdir/mriqc_wf/funcMRIQC/fMRI_HMC/_in_file_..data..sub-02..func..sub-02_task-rhymejudgment_bold.nii.gz/gen_ref/slice.nii.gz" }, "4159642": { "pcpu": 5.7, "pmem": 0.0, "rss": 11001856, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175780-04:00", "etime": "00:45", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(8)" }, "4159771": { "pcpu": 4.4, "pmem": 0.0, "rss": 10956800, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175909-04:00", "etime": "00:39", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(8)" }, "4159226": { "pcpu": 109.0, "pmem": 0.0, "rss": 119955456, "vsz": 365039616, "timestamp": "2024-09-19T14:24:03.175198-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159228": { "pcpu": 100.0, "pmem": 0.0, "rss": 282193920, "vsz": 703242240, "timestamp": "2024-09-19T14:24:03.175242-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159610": { "pcpu": 2.6, "pmem": 0.0, "rss": 11001856, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175758-04:00", "etime": "00:45", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(8)" }, "4159738": { "pcpu": 3.8, "pmem": 0.0, "rss": 11001856, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175801-04:00", "etime": "00:43", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(8)" }, "4159230": { "pcpu": 104.0, "pmem": 0.0, "rss": 228229120, "vsz": 712790016, "timestamp": "2024-09-19T14:24:03.175286-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159932": { "pcpu": 6.2, "pmem": 0.0, "rss": 10895360, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175929-04:00", "etime": "00:27", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(8)" }, "4159934": { "pcpu": 0.0, "pmem": 0.0, "rss": 761856, "vsz": 4620288, "timestamp": "2024-09-19T14:24:03.175950-04:00", "etime": "00:26", "cmd": "S /bin/sh -c 3dvolreg -Fourier -twopass -1Dfile sub-02_task-rhymejudgment_bold_valid.1D -1Dmatrix_save sub-02_task-rhymejudgment_bold_valid.aff12.1D -prefix sub-02_task-rhymejudgment_bold_valid_volreg.nii.gz -base /workdir/mriqc_wf/funcMRIQC/fMRI_HMC/_in_file_..data..sub-02..func..sub-02_task-rhymejudgment_bold.nii.gz/gen_ref/ref_bold.nii.gz -zpad 4 -maxdisp1D sub-02_task-rhymejudgment_bold_valid_md.1D /workdir/mriqc_wf/funcMRIQC/fMRI_HMC/_in_file_..data..sub-02..func..sub-02_task-rhymejudgment_bold.nii.gz/motion_correct/sub-02_task-rhymejudgment_bold_valid.nii.gz" }, "4159935": { "pcpu": 101.0, "pmem": 0.0, "rss": 56315904, "vsz": 97992704, "timestamp": "2024-09-19T14:24:03.175972-04:00", "etime": "00:26", "cmd": "R 3dvolreg -Fourier -twopass -1Dfile sub-02_task-rhymejudgment_bold_valid.1D -1Dmatrix_save sub-02_task-rhymejudgment_bold_valid.aff12.1D -prefix sub-02_task-rhymejudgment_bold_valid_volreg.nii.gz -base /workdir/mriqc_wf/funcMRIQC/fMRI_HMC/_in_file_..data..sub-02..func..sub-02_task-rhymejudgment_bold.nii.gz/gen_ref/ref_bold.nii.gz -zpad 4 -maxdisp1D sub-02_task-rhymejudgment_bold_valid_md.1D /workdir/mriqc_wf/funcMRIQC/fMRI_HMC/_in_file_..data..sub-02..func..sub-02_task-rhymejudgment_bold.nii.gz/motion_correct/sub-02_task-rhymejudgment_bold_valid.nii.gz" }, "4159754": { "pcpu": 102.0, "pmem": 0.0, "rss": 76070912, "vsz": 95371264, "timestamp": "2024-09-19T14:24:03.175844-04:00", "etime": "00:42", "cmd": "R N4BiasFieldCorrection -d 3 --input-image /workdir/mriqc_wf/anatMRIQC/_in_file_..data..sub-02..anat..sub-02_T1w.nii.gz/conform/sub-02_T1w_conformed.nii.gz --output [ sub-02_T1w_conformed_corrected.nii.gz, sub-02_T1w_conformed_bias.nii.gz ] -r" }, "4158281": { "pcpu": 2.8, "pmem": 0.0, "rss": 19939328, "vsz": 1276940288, "timestamp": "2024-09-19T14:24:03.174912-04:00", "etime": "01:00", "cmd": "Ssl Singularity runtime parent" }, "4158932": { "pcpu": 83.7, "pmem": 0.0, "rss": 227446784, "vsz": 788496384, "timestamp": "2024-09-19T14:23:12.163871-04:00", "etime": "00:05", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4158306": { "pcpu": 87.7, "pmem": 0.0, "rss": 230993920, "vsz": 865636352, "timestamp": "2024-09-19T14:24:03.174991-04:00", "etime": "01:00", "cmd": "R /usr/local/miniconda/bin/python /usr/local/miniconda/bin/mriqc /data /out participant --participant-label 02 -w /workdir --no-sub" }, "4159074": { "pcpu": 10.7, "pmem": 0.0, "rss": 11018240, "vsz": 26898432, "timestamp": "2024-09-19T14:23:12.163878-04:00", "etime": "00:02", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(9)" }, "4159084": { "pcpu": 4.9, "pmem": 0.0, "rss": 10915840, "vsz": 26898432, "timestamp": "2024-09-19T14:23:13.197145-04:00", "etime": "00:01", "cmd": "S /usr/local/miniconda/bin/python -s -Wignore -c from multiprocessing.semaphore_tracker import main;main(11)" }, "4159220": { "pcpu": 104.0, "pmem": 0.0, "rss": 119717888, "vsz": 365023232, "timestamp": "2024-09-19T14:24:03.175070-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159221": { "pcpu": 104.0, "pmem": 0.0, "rss": 216580096, "vsz": 628342784, "timestamp": "2024-09-19T14:24:03.175091-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159222": { "pcpu": 104.0, "pmem": 0.0, "rss": 119906304, "vsz": 365027328, "timestamp": "2024-09-19T14:24:03.175114-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159223": { "pcpu": 104.0, "pmem": 0.0, "rss": 213327872, "vsz": 622346240, "timestamp": "2024-09-19T14:24:03.175134-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159224": { "pcpu": 109.0, "pmem": 0.0, "rss": 119795712, "vsz": 364994560, "timestamp": "2024-09-19T14:24:03.175155-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159225": { "pcpu": 100.0, "pmem": 0.0, "rss": 119623680, "vsz": 365035520, "timestamp": "2024-09-19T14:24:03.175175-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4158842": { "pcpu": 4.2, "pmem": 0.0, "rss": 10948608, "vsz": 26898432, "timestamp": "2024-09-19T14:24:03.175023-04:00", "etime": "00:57", "cmd": "S /usr/local/miniconda/bin/python -s -c from multiprocessing.semaphore_tracker import main;main(7)" }, "4159227": { "pcpu": 104.0, "pmem": 0.0, "rss": 229507072, "vsz": 720797696, "timestamp": "2024-09-19T14:24:03.175221-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4158844": { "pcpu": 63.6, "pmem": 0.0, "rss": 232763392, "vsz": 912482304, "timestamp": "2024-09-19T14:23:13.197122-04:00", "etime": "00:07", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159229": { "pcpu": 104.0, "pmem": 0.0, "rss": 119898112, "vsz": 365031424, "timestamp": "2024-09-19T14:24:03.175264-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" }, "4159753": { "pcpu": 0.0, "pmem": 0.0, "rss": 774144, "vsz": 4620288, "timestamp": "2024-09-19T14:24:03.175823-04:00", "etime": "00:42", "cmd": "S /bin/sh -c N4BiasFieldCorrection -d 3 --input-image /workdir/mriqc_wf/anatMRIQC/_in_file_..data..sub-02..anat..sub-02_T1w.nii.gz/conform/sub-02_T1w_conformed.nii.gz --output [ sub-02_T1w_conformed_corrected.nii.gz, sub-02_T1w_conformed_bias.nii.gz ] -r" }, "4159231": { "pcpu": 100.0, "pmem": 0.0, "rss": 306819072, "vsz": 797450240, "timestamp": "2024-09-19T14:24:03.175307-04:00", "etime": "00:49", "cmd": "R /usr/local/miniconda/bin/python -s -c from multiprocessing.forkserver import main; main(7, 9, ['__main__'], **{'sys_path': ['/usr/local/miniconda/bin', '/usr/local/miniconda/lib/python37.zip', '/usr/local/miniconda/lib/python3.7', '/usr/local/miniconda/lib/python3.7/lib-dynload', '/usr/local/miniconda/lib/python3.7/site-packages']})" } }, "totals": { "pmem": 0.0, "pcpu": 3438.4, "rss": 5395185664, "vsz": 17575694336 }, "averages": { "rss": 4250917711.448277, "vsz": 14342254309.51724, "pmem": 0.0, "pcpu": 723.3931034482758, "num_samples": 58 } } ```

[asmacdo]

and indeed, if I go through and add the peaks of each pid as listed above, we get an even higher number

In [25]: for pid, stats in data['processes'].items():
    ...:     for k, v in stats.items():
    ...:         if k == 'pcpu':
    ...:             tot += v
    ...: 

In [26]: 

In [26]: tot
Out[26]: 4000.2999999999993

[yarikoptic]

Please wrap long dumps into <details> next time. So was that on Typhon? If so, let's consider ok and proceed

[asmacdo]

Yes on typhon