Kafka Publisher publishes data stream fields in a topic in inconstant order

[FabriceHansen]

Apache StreamPipes version

dev (current development state)

Affected StreamPipes components

Backend

What happened?

I am publishing a data stream to Kafka for a live Kafka data source in Grafana. As Streampipes Kafka Publisher publishes data stream fields in a topic in inconstant order, the colours of Grafana Time Series panel series are continuously changing.

How to reproduce?

create a pipeline with a data stream, a projection processor to select fields and remove timestamp. Connect it to a Kafka Publisher Data Sink. Define Topic Name, Use kafka default Kafka Broker and port 9092. Select "Unauthenticated Plain" option. In Grafana, install Kafka Plugin and create a new Kafka connection using "kafka:9092" parameters. Create a new dashboard with a Time Series Panel. Add a query with Kafka Data Source with the same topic name as defined in SP Kafka Publisher Sink.

Expected behavior

I would expect to receive all data stream fields in the same order for every Kafka publishing.

Additional technical information

No response

Are you willing to submit a PR?

None

[dominikriemer]

Hi @FabriceHansen I wouldn't classify this as a bug: StreamPipes internally uses a Map data structure to represent events, and the Kafka sink serializes messages as a JSON object. Both representations don't guarantee order. This could be only adressed in StreamPipes by changing the output of the Kafka sink to a list instead of an object. Isn't there a way in Grafana to map a key to a series?

[FabriceHansen]

Hi @dominikriemer, found a way to force order in Grafana Data Transformation with Organize fields by name. Nevertheless, it would be nice to have the option directly in the Kafka publisher sink. I would agree to change from bug to improvement.