Open JulienPeloton opened 2 years ago
Note: management of groupid
is not great for Spark 2.4.x. and we are forced to use spark-kafka-source
as prefix for the groupid...
Spark 3.x allows to set it though: https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#kafka-specific-configurations
We might want to test this with kubernetes then.
Describe the issue
The integration testing deployment of the Rubin alert stream is now available. Information for the connection can be found at https://github.com/lsst-dm/sample_alert_info/blob/main/doc/alert_stream_integration_endpoint.md
Concretely, this means for us:
bin/stream2raw.py
(perhaps another script?).fink_broker/sparkUtils.py