astrolabsoftware / fink-broker

Astronomy Broker based on Apache Spark
https://fink-broker.org
Apache License 2.0
68 stars 13 forks source link

[LSST] Set up a consumer for Rubin test stream #501

Open JulienPeloton opened 2 years ago

JulienPeloton commented 2 years ago

Describe the issue

The integration testing deployment of the Rubin alert stream is now available. Information for the connection can be found at https://github.com/lsst-dm/sample_alert_info/blob/main/doc/alert_stream_integration_endpoint.md

Concretely, this means for us:

JulienPeloton commented 2 years ago

Note: management of groupid is not great for Spark 2.4.x. and we are forced to use spark-kafka-source as prefix for the groupid...

Spark 3.x allows to set it though: https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#kafka-specific-configurations

We might want to test this with kubernetes then.