Potentially auto-tune comm aggregators to system configuration

Bears-R-Us / arkouda

Arkouda (αρκούδα): Interactive Data Analytics at Supercomputing Scale :bear:

Other

236 stars 87 forks source link

Potentially auto-tune comm aggregators to system configuration #726

Open mhmerrill opened 3 years ago

mhmerrill commented 3 years ago

auto-tune comm aggregators to system configuration

since buffers are sized to num-cores x num-locales
buffers will spill out of cache potentially at larger node counts
possibly tune at server startup
this will effect many server operations like sort, int-indexed gather/scatter, boolean-indexed expand/compress, etc.

ronawho commented 3 years ago

I've been thinking about this off and on and debating if this should/could be done at server startup or if it should be an external process to reduce arkouda start up times. Where I think the tradeoffs are that if this is done in arkouda at startup you're tuning for the current numLocales and system at the cost of startup time, but an offline process (or something that caches values) your results could become stale but won't impact start up time.