open2c / cooltools

The tools for your .cool's
MIT License
140 stars 51 forks source link

Update pool decorator, and multiprocessing #539

Closed Yaoyx closed 1 month ago

Yaoyx commented 1 month ago

535

536

gfudenberg commented 1 month ago

@dmitrymyl I just invited you as a collaborator for the repo-- let us know if this resolves the issues you encountered!

dmitrymyl commented 1 month ago

I checked this version against my profiling notebook -- now obs/exp pups are calculated fast and one can also supply custom map functors (as intended). So, this PR resolves both issues.

Timings:

Pileup kind nproc cooltools (default) cooltools x multiprocess cooltools x multiprocessing cooltools x concurrent.futures coolpuppy (default)
OBS 1 9.8 s 10.4 s 10.4 s 10.1 s 8.93 s
OBS 8 2.71 s 2.74 s 2.74 s 2.69 s 2.45 s
OBSEXP 1 11.7 s 13.8 s 12.6 s 14.6 s 14.6 s
OBSEXP 8 5.37 s 19.7 s 4.48 s 5.06 s 9.4 s