skyportal / gwemopt

Gravitational-wave Electromagnetic Optimization
GNU General Public License v3.0
19 stars 34 forks source link

Parallelization optimization #147

Closed tomas-cabrera closed 1 year ago

tomas-cabrera commented 1 year ago

Added some arguments to joblib.Parallel() calls in moc.py and segments.py to for faster runtimes.

After some benchmarking, it was determined that a function of the number of tasks and the number of cores was a reasonable way to determining the batch size (better than using a fixed batch size for all core counts, or joblib's dynamic batch size, which is too slow for large number of cores). The batch size function in this commit scales well to 16 cores, and the fastest performance was recorded when running on 32 cores (testing extended to 64 and 128 core runs, but slower overall times were observed at these scales). Using the multiprocessing backend (instead of the default loky) also bought a small improvement. Overall speedup due to this improvement was ~3-14x when compared to the previous version, with the highest impacts at larger core numbers. This update was tested with both the default, 13,902-pointing DECam tessellation, and a custom DECam tessellation of 45,713 pointings.

This commit also adds a missing argument in a function call in segments.py, and changes a matplotlib.patches.PathPatch keyword in pixels.py to alleviate matplotlib warning messages.

mcoughlin commented 1 year ago

@tomas-cabrera could you please lint your files? We do this with:

pre-commit run --files path-to-files-you-edit

tomas-cabrera commented 1 year ago

Ok, I think that's done. Let me know if everything looks good, first time using pre-commit!