Currently, whenever an alert passes one or more filters in Kowalski, we end up with a lot of API calls sent by Kowalski TO SkyPortal. This means that SkyPortal's API is very busy during the night and that its load is impacted by the filters themselves (the looser the filters, the more we add latency to the system). This also means that Kowalski is affected by any slowdowns or crashes of SkyPortal, which slows down alert stream processing.
Since SkyPortal has all of the code to process the alert data already, it would make a lot more sense if Fritz/SkyPortal would be the one reading from Kowalski at its own pace and adding the alerts that pass any filter to the database (along with other data products, and perform actions like automated triggering).
This is particularly true for any filters with an autosaving or auto triggering pipeline. These add extra latency as they run expensive queries to SkyPortal, while already offloading most of the validation to SkyPortal.
We could have Kowalski output the results of its filters to one or more Kafka topics. On top of improving if not completely removing latency issues, we end up with topics that not one by many SkyPortal instances could read from, and can resume reading from if they are not up during the night.
This could take advantage of this plugin/extension system, where we would have a Kowalski plugin that Fritz could use and configure in its config.
Currently, whenever an alert passes one or more filters in Kowalski, we end up with a lot of API calls sent by Kowalski TO SkyPortal. This means that SkyPortal's API is very busy during the night and that its load is impacted by the filters themselves (the looser the filters, the more we add latency to the system). This also means that Kowalski is affected by any slowdowns or crashes of SkyPortal, which slows down alert stream processing.
Since SkyPortal has all of the code to process the alert data already, it would make a lot more sense if Fritz/SkyPortal would be the one reading from Kowalski at its own pace and adding the alerts that pass any filter to the database (along with other data products, and perform actions like automated triggering).
This is particularly true for any filters with an autosaving or auto triggering pipeline. These add extra latency as they run expensive queries to SkyPortal, while already offloading most of the validation to SkyPortal.
We could have Kowalski output the results of its filters to one or more Kafka topics. On top of improving if not completely removing latency issues, we end up with topics that not one by many SkyPortal instances could read from, and can resume reading from if they are not up during the night.
This could take advantage of this plugin/extension system, where we would have a Kowalski plugin that Fritz could use and configure in its config.