Open FabriceHansen opened 3 days ago
Hi @FabriceHansen I wouldn't classify this as a bug: StreamPipes internally uses a Map
data structure to represent events, and the Kafka sink serializes messages as a JSON
object. Both representations don't guarantee order. This could be only adressed in StreamPipes by changing the output of the Kafka sink to a list instead of an object.
Isn't there a way in Grafana to map a key to a series?
Hi @dominikriemer, found a way to force order in Grafana Data Transformation with Organize fields by name. Nevertheless, it would be nice to have the option directly in the Kafka publisher sink. I would agree to change from bug to improvement.
Apache StreamPipes version
dev (current development state)
Affected StreamPipes components
Backend
What happened?
I am publishing a data stream to Kafka for a live Kafka data source in Grafana. As Streampipes Kafka Publisher publishes data stream fields in a topic in inconstant order, the colours of Grafana Time Series panel series are continuously changing.
How to reproduce?
create a pipeline with a data stream, a projection processor to select fields and remove timestamp. Connect it to a Kafka Publisher Data Sink. Define Topic Name, Use kafka default Kafka Broker and port 9092. Select "Unauthenticated Plain" option. In Grafana, install Kafka Plugin and create a new Kafka connection using "kafka:9092" parameters. Create a new dashboard with a Time Series Panel. Add a query with Kafka Data Source with the same topic name as defined in SP Kafka Publisher Sink.
Expected behavior
I would expect to receive all data stream fields in the same order for every Kafka publishing.
Additional technical information
No response
Are you willing to submit a PR?
None