Kristina-Pianykh / flink-multinode

Other
0 stars 0 forks source link

[fix]: Deduplicate output stream when merged with input stream #2

Closed Kristina-Pianykh closed 1 month ago

Kristina-Pianykh commented 3 months ago

For now, the output stream produced by the processing of the queries on a node can contain the same events as were originally part of the input stream. This results in sending duplicate events.

https://github.com/Kristina-Pianykh/flink-multinode/blob/98b7e622c3e7aaae5f616f06c0086f2b46494903/java-cep/src/main/java/com/huberlin/DataStreamJob.java#L161-L174

[EDIT]: important to do that before forwarding events further (to keep the transmission rate low)