For a typical chunk processing steps 7, 8, 9 take ~6 minutes but without copy-to-tmp only 1 minute. If we just use 3 nodes the processing time can be reduced from 240 minutes (4 hours) to 40 minutes (0.66 hours)
run_07: 15 jobs 5*6=30 5*1=5 (each job 9 tasks) (each tasks requires 15 GB input (240 files, 80 of which are big (0.15GB))
Lets
run_08: 70 jobs 24*6=140 24*1=24 (each job 9 tasks) (each tasks requires about 7.5 GB (120 files, 40 of which are big (0.15 GB)). This task is memory intensive and the number of tasks per job is memory-limited (1 task about 18 GB)
run_09: 35 jobs. 12*6=72 12*1=12 (each job 18 task) (each task requires 7.5 GB (90 files of which 30 are big (0.25 GB))
242 41
For a typical chunk processing steps 7, 8, 9 take ~6 minutes but without copy-to-tmp only 1 minute. If we just use 3 nodes the processing time can be reduced from 240 minutes (4 hours) to 40 minutes (0.66 hours)
KokoxiliChunk30SenDT121