Surge of load when running postprocessing

[dwimjpurnomo]

When I run the postprocessing the load average is so high that it reach 10,000. The available cores is 48. It correspond to the number of files to be converted from raw to tif, it is 50,000 files. It does not matter if we use Hamada or UCB model. I am thinking to limit the usage of computational power in the postprocessing. What are the options available?

[lautenberger]

@dwimjpurnomo are you looping over a large number of files, something like this?

for f in *.bil; do
   do_something_to $f &
done

If so, this how you can ensure that you launch only the same number of parallel processes as your number of CPU cores:

NPARALLEL=`grep ^cpu\\scores /proc/cpuinfo | uniq |  awk '{print $4}'`
N=0
for f in *.bil; do
   do_something_to $f  &
   let "N = N + 1"
   if [ "$N" = "$NPARALLEL" ]; then
      wait
      N=0
   fi
done
wait

There are also fancier ways but this works pretty well, I use something similar when processing a large number of files. Note you probably wouldn't want to do this if reading from a NAS because i/o will become the bottleneck but if you're working from an SSD - particularly NVMe - i/o bandwidth shouldn't get saturated.

[dwimjpurnomo]

Thank you, I will try this. Yes, the loop is then for 50,000 files

[dwimjpurnomo]

It works perfectly. Thanks