Requires large chunk of memory when having many sources

icecube / skyllh

https://icecube.github.io/skyllh/

GNU General Public License v3.0

11 stars 5 forks source link

Requires large chunk of memory when having many sources #161

Open martwo opened 1 year ago

martwo commented 1 year ago

This lines brakes when using many sources (400+) due to large memory allocation of 20+GB. Maybe use int32 instead of int64!? https://github.com/icecube/skyllh/blob/e99102e3dc3859f6e1460320e8b226e600dea739/skyllh/core/event_selection.py#L577

martwo commented 1 year ago

It is not possible to specify the dtype of the resulting idxs array. Also np.nonzero(a) does not support to specify the dtype of the resulting array :( One might have to write a batched version of np.argwhere and batch chunks of sources.

martwo commented 1 year ago

The issue arises for the pre_event_selection_method of the background generation method, when O(10 million) MC events are selected for all the sources.