search

con / duct

A helper to run a command, capture stdout/stderr and details about running
MIT License
1 stars 1 forks source link

check what duct is "busy with" #60

Open yarikoptic opened 3 weeks ago

yarikoptic commented 3 weeks ago

ATM running a singularity container through duct shows 2-3% CPU of duct. I wonder what it is doing. Would be worth checking with py-spy etc some of its runs to see what it spends time on -- even though just a little, given how little it needs to do I would expect it to be even lower

yoh      1622935  0.1  0.0 271356 39816 pts/13   Sl+  10:58   0:00       /usr/bin/python3 /usr/bin/datalad run -m Doing trial run with bleeding edge 0.0.5 duct singularity run -B /mnt/DATA/data/yoh/1076_spacetop:/mnt/DATA/data/yoh/1076_spacetop:ro -B /mnt/DATA/data/yoh/1076_spacetop/derivatives:/mnt/DATA/data/yoh/1076_spacetop/derivatives:rw -B /mnt/DATA/data/yoh/1076_spacetop/scratch:/mnt/DATA/data/yoh/1076_spacetop/scratch:rw -B /mnt/DATA/data/yoh/1076_spacetop/../dsst-defacing-pipeline/src:/opt/dsst-defacing-pipeline:ro --scratch /mnt/DATA/data/yoh/1076_spacetop/scratch --pwd /mnt/DATA/data/yoh/1076_spacetop --net --network none ../dsst-defacing-pipeline/dsst-defacing-pipeline-0.0.5.sif -p 0001 -s 01 -- . derivatives/dss-defacing-0.0.5-1
yoh      1623044  0.0  0.0   2580   904 pts/13   S+   10:58   0:00         /bin/sh -c duct singularity run -B /mnt/DATA/data/yoh/1076_spacetop:/mnt/DATA/data/yoh/1076_spacetop:ro -B /mnt/DATA/data/yoh/1076_spacetop/derivatives:/mnt/DATA/data/yoh/1076_spacetop/derivatives:rw -B /mnt/DATA/data/yoh/1076_spacetop/scratch:/mnt/DATA/data/yoh/1076_spacetop/scratch:rw -B /mnt/DATA/data/yoh/1076_spacetop/../dsst-defacing-pipeline/src:/opt/dsst-defacing-pipeline:ro --scratch /mnt/DATA/data/yoh/1076_spacetop/scratch --pwd /mnt/DATA/data/yoh/1076_spacetop --net --network none ../dsst-defacing-pipeline/dsst-defacing-pipeline-0.0.5.sif -p 0001 -s 01 -- . derivatives/dss-defacing-0.0.5-1
yoh      1623046  2.3  0.0 241476 14000 pts/13   Sl+  10:58   0:15           /mnt/DATA/data/yoh/dsst-defacing-pipeline/venvs/dev3/bin/python3 /mnt/DATA/data/yoh/dsst-defacing-pipeline/venvs/dev3/bin/duct singularity run -B /mnt/DATA/data/yoh/1076_spacetop:/mnt/DATA/data/yoh/1076_spacetop:ro -B /mnt/DATA/data/yoh/1076_spacetop/derivatives:/mnt/DATA/data/yoh/1076_spacetop/derivatives:rw -B /mnt/DATA/data/yoh/1076_spacetop/scratch:/mnt/DATA/data/yoh/1076_spacetop/scratch:rw -B /mnt/DATA/data/yoh/1076_spacetop/../dsst-defacing-pipeline/src:/opt/dsst-defacing-pipeline:ro --scratch /mnt/DATA/data/yoh/1076_spacetop/scratch --pwd /mnt/DATA/data/yoh/1076_spacetop --net --network none ../dsst-defacing-pipeline/dsst-defacing-pipeline-0.0.5.sif -p 0001 -s 01 -- . derivatives/dss-defacing-0.0.5-1
yoh      1623049  0.0  0.0 1247012 16620 ?       Ssl  10:58   0:00             Singularity runtime parent
yoh      1623076  0.0  0.0  19740 16584 ?        S    10:58   0:00               python /opt/dsst-defacing-pipeline/run.py -p 0001 -s 01 -- . derivatives/dss-defacing-0.0.5-1
yoh      1623098  0.0  0.0   4040  3056 ?        S    10:58   0:00                 /bin/tcsh /usr/bin/@afni_refacer_run -input /mnt/DATA/data/yoh/1076_spacetop/sub-0001/ses-01/anat/sub-0001_ses-01_acq-MPRAGEXp3X08mm_T1w.nii.gz -mode_deface -no_clean -prefix /mnt/DATA/data/yoh/1076_spacetop/derivatives/dss-defacing-0.0.5-1/bids_defaced/sub-0001/ses-01/anat/sub-0001_ses-01_acq-MPRAGEXp3X08mm_T1w
yoh      1674377  100  0.0 326992 320464 ?       R    11:10   0:01                   3dcalc -a tmp.00.INPUT.nii -c tmp.05.sh_t2a_thr.nii[1] -expr a*not(bool(c)) -prefix tmp.99.result.deface.nii -datum float -ISOLA

but I could be wrong.

asmacdo commented 1 week ago

This bugfix helps https://github.com/con/duct/commit/b6c9219bc3a20f781f321f906396ba0a25bc5250

Here's some fun dogfooding :)

$ duct duct sleep 100
duct is executing duct sleep 100...
Log files will be written to .duct/logs/2024.06.21T13.06.49-1223368_
duct is executing sleep 100...
Log files will be written to .duct/logs/2024.06.21T13.06.49-1223371_
Exit Code: 0
Command: sleep 100
Log files location: .duct/logs/2024.06.21T13.06.49-1223371_
Wall Clock Time: 100.003 sec sec
Memory Peak Usage (RSS): 1920 KiB
Memory Average Usage (RSS): 1920.000 KiB
Virtual Memory Peak Usage (VSZ): 221584 KiB
Virtual Memory Average Usage (VSZ): 221584.000 KiB
Memory Peak Percentage: 0.0%
Memory Average Percentage: 0.000%
CPU Peak Usage: 0.0%
Average CPU Usage: 0.000%
Samples Collected: 98
Reports Written: 2

Exit Code: 0
Command: duct sleep 100
Log files location: .duct/logs/2024.06.21T13.06.49-1223368_
Wall Clock Time: 100.073 sec sec
Memory Peak Usage (RSS): 17920 KiB
Memory Average Usage (RSS): 17196.735 KiB
Virtual Memory Peak Usage (VSZ): 682564 KiB
Virtual Memory Average Usage (VSZ): 627538.367 KiB
Memory Peak Percentage: 0.0%
Memory Average Percentage: 0.000%
CPU Peak Usage: 4.0%
Average CPU Usage: 1.658%
Samples Collected: 98
Reports Written: 2

Average CPU Usage: 1.658% is an improvement, but still using a lot more power than we probably need.