AmpelAstro / Ampel-nuclear

Ampel units to analyze nuclear events (e.g. searching for Tidal Disruption Events)
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Incremental processing #27

Closed jvansanten closed 1 year ago

jvansanten commented 1 year ago

When handed thousands of TransientViews, the T3 process can consume multiple tens of GB of memory and potentially be terminated by the OOM killer. Avoid this by consuming TransientViews only as needed, and flushing the dropbox upload cache occasionally (every 100 MB by default).