I've found that eliminating dask multiprocessing can speed up the inner_loop function's main loop by around 25x (each iteration: ~0.75s without dask, ~20s with dask on my machine), probably due to high startup times. This might not apply to every machine.
Now, I can fail to find the steady-state 25x faster! :smile:
I've found that eliminating
dask
multiprocessing can speed up theinner_loop
function's main loop by around 25x (each iteration: ~0.75s without dask, ~20s with dask on my machine), probably due to high startup times. This might not apply to every machine.Now, I can fail to find the steady-state 25x faster! :smile:
@rickecon @jdebacker @jpycroft @MaxGhenis