Saving time without copy-to-tmp ?

[falkamelung]

For a typical chunk processing steps 7, 8, 9 take ~6 minutes but without copy-to-tmp only 1 minute. If we just use 3 nodes the processing time can be reduced from 240 minutes (4 hours) to 40 minutes (0.66 hours)

run_07: 15 jobs     5*6=30       5*1=5            (each job 9 tasks) (each tasks requires 15 GB  input (240 files, 80 of which are big (0.15GB))
Lets
run_08: 70 jobs    24*6=140    24*1=24       (each job 9 tasks) (each tasks requires about 7.5 GB (120 files, 40 of which are big (0.15 GB)).  This task is memory intensive and the number of tasks per job is memory-limited (1 task about 18 GB)

run_09: 35 jobs.   12*6=72     12*1=12          (each job 18 task) (each task requires 7.5 GB (90 files of which 30 are big (0.25 GB)) 
                       242          41

KokoxiliChunk30SenDT121

  NNodes  Timelimit   Reserved    Elapsed                        JobName      State 
-------- ---------- ---------- ---------- ------------------------------ ---------- 
       1   00:14:00   00:15:58   00:01:02 run_07_merge_reference_second+  COMPLETED 
       1                         00:01:02                          batch  COMPLETED 

  NNodes  Timelimit   Reserved    Elapsed                        JobName      State 
-------- ---------- ---------- ---------- ------------------------------ ---------- 
       1   00:15:00   00:01:55   00:00:42  run_08_generate_burst_igram_0  COMPLETED 
       1                         00:00:42                          batch  COMPLETED 

  NNodes  Timelimit   Reserved    Elapsed                        JobName      State 
-------- ---------- ---------- ---------- ------------------------------ ---------- 
       1   00:14:00   00:00:13   00:00:41     run_09_merge_burst_igram_2  COMPLETED 
       1                         00:00:41                          batch  COMPLETED 

  NNodes  Timelimit   Reserved    Elapsed                        JobName      State 
-------- ---------- ---------- ---------- ------------------------------ ---------- 
       1   00:08:00   00:01:13   00:00:45      run_10_filter_coherence_1  COMPLETED 
       1                         00:00:45                          batch  COMPLETED 
  NNodes  Timelimit   Reserved    Elapsed                        JobName      State 
-------- ---------- ---------- ---------- ------------------------------ ---------- 
       1   00:03:00   00:00:02   00:01:17                run_11_unwrap_1  COMPLETED 
       1                         00:01:17                          batch  COMPLETED 

run_07_merge_reference_secondary_slc_1        1   00:14:00   00:01:17   00:06:55   00:00:14
run_08_generate_burst_igram_2           1   00:15:00   00:01:52   00:06:19   00:00:13
run_09_merge_burst_igram_1              1   00:14:00   00:16:51   00:06:56   00:00:14
run_10_filter_coherence_1               1   00:08:00   00:04:05   00:00:53   00:00:01
run_11_unwrap_1                         1   00:12:00   00:02:59   00:01:21   00:00:02