Added some arguments to joblib.Parallel() calls in moc.py and segments.py to for faster runtimes.
After some benchmarking, it was determined that a function of the number of tasks and the number of cores was a reasonable way to determining the batch size (better than using a fixed batch size for all core counts, or joblib's dynamic batch size, which is too slow for large number of cores). The batch size function in this commit scales well to 16 cores, and the fastest performance was recorded when running on 32 cores (testing extended to 64 and 128 core runs, but slower overall times were observed at these scales). Using the multiprocessing backend (instead of the default loky) also bought a small improvement. Overall speedup due to this improvement was ~3-14x when compared to the previous version, with the highest impacts at larger core numbers. This update was tested with both the default, 13,902-pointing DECam tessellation, and a custom DECam tessellation of 45,713 pointings.
This commit also adds a missing argument in a function call in segments.py, and changes a matplotlib.patches.PathPatch keyword in pixels.py to alleviate matplotlib warning messages.
Added some arguments to
joblib.Parallel()
calls inmoc.py
andsegments.py
to for faster runtimes.After some benchmarking, it was determined that a function of the number of tasks and the number of cores was a reasonable way to determining the batch size (better than using a fixed batch size for all core counts, or
joblib
's dynamic batch size, which is too slow for large number of cores). The batch size function in this commit scales well to 16 cores, and the fastest performance was recorded when running on 32 cores (testing extended to 64 and 128 core runs, but slower overall times were observed at these scales). Using themultiprocessing
backend (instead of the defaultloky
) also bought a small improvement. Overall speedup due to this improvement was ~3-14x when compared to the previous version, with the highest impacts at larger core numbers. This update was tested with both the default, 13,902-pointing DECam tessellation, and a custom DECam tessellation of 45,713 pointings.This commit also adds a missing argument in a function call in
segments.py
, and changes amatplotlib.patches.PathPatch
keyword inpixels.py
to alleviatematplotlib
warning messages.