[SPRINT] ELAsTiCC - work on TOM Toolkit/Marshal system on CC @ IN2P3

[gnarayan]

ELAsTiCC - work on TOM Toolkit/Marshal system on CC @ IN2P3

[One sentence description]

Contacts: Gautham, Alex Kim, Johann Cohen-Tanugi Day/Time: All - focused at computing resources at IN2P3 Main communication channel: #desc-alerts-topical-team GitHub repo: https://github.com/LSSTDESC/tom_desc

Goals and deliverable

Make sure streams from all brokers in ELAsTiCC challenge (ALeRCE, AMPEL, ANTARES, Fink, LASAIR, PGB) can be listened to using TOM Toolkit Save classified alerts to (minimally) disk, or (ideally) a relational DB Start working on visualizations/queries to evaluate performance for each broker/classifier combo

Resources and skills needed

Python/SQL Will need CC account

Detailed description

DESC wants a dashboard for LSST operations to view the alerts from brokers and select high-confidence SNe Ia. There is no infrastructure for this yet. This is the start of us building what is required. We will use off-the-shelf resources as far as possible - TOM Toolkit/Sky Portal/YSE PZ.

[JulienPeloton]

There are already TOM packages available for some of the brokers:

• ALeRCE: https://github.com/TOMToolkit/tom_base/blob/main/tom_alerts/brokers/alerce.py
• ANTARES: https://github.com/TOMToolkit/tom_antares
• Fink: https://github.com/TOMToolkit/tom_fink
• Lasair: https://github.com/TOMToolkit/tom_base/blob/main/tom_alerts/brokers/lasair.py

Some of them are not syncing data "live". E.g. in Fink, the TOM is only linked to the Fink REST API that exposes processed live data at the end of each night. So just to confirm, we want all brokers to be listened to using TOM, and in real-time?

[troyraen]

Pitt-Google now has proof-of-concept demos at: https://github.com/mwvgroup/tom_pittgoogle

[gnarayan]

@JulienPeloton - I don't think it has to be in real-time to all the brokers. We know (as @AlexGKim and @rbiswas4 wrote in the original DESC LONI) that DESC doesn't actually need alerts in real-time and can live with 1 day latency. Certainly there are going to be metrics that measure latency, and the end-of-night processing model of FINK will have an impact there, but again we're not doing this as a competition in the same way and I think a latency number is only useful in as much as it's interpreted as "below DESC requirement."

I'd say the next thing we should focus on if the interface is set is a) setting up a test broadcast of alerts with your scripts as we discussed today b) writing some web dashboard to allow us monitor the alerts from each broker after they are stored

The second needs some kind of tutorial on the TOM system and how it saves alerts and how we can access them programmatically from some combination of @AlexGKim @troyraen and you.