thouska / spotpy

A Statistical Parameter Optimization Tool
https://spotpy.readthedocs.io/en/latest/
MIT License
254 stars 152 forks source link

simulation timeout thread is started unconditionally, leading to unnecessary overhead #253

Closed wookietreiber closed 4 years ago

wookietreiber commented 4 years ago

We realized that for very short (few seconds) simulations, there is considerable overhead:

$ time strace -c -f -p 5634
...
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 65.13    0.006429           2      2997       699 futex
 32.42    0.003200          33        96           madvise
  1.62    0.000160           1        96           clone
  0.83    0.000082           0        96           set_robust_list
  0.00    0.000000           0         2           write
------ ----------- ----------- --------- --------- ----------------
100.00    0.009871                  3287       699 total

real    0m3.984s
user    0m0.027s
sys    0m0.070s

There's a lot going on in thread creation (clone) and coordination (futex) that is not necessary, especially not for a single-core run. This is due to def simulate in spotpy/algorithms/_algorithm.py.

My proposal is to only spawn the thread if self.sim_timeout is set to avoid the above-mentioned overhead. I'll prepare a pull request.

thouska commented 4 years ago

Thanks for testing this! Indeed this whole thread creation section might be not necessary anymore, as it was only included as a workaround for a python2/3 compability issue. Your proposal sounds good, I would highly appreciate your pull-request on this.

wookietreiber commented 4 years ago

@thouska Great, please review #254.