Avoid duplication of events in an existing Kafka topic when using a KafkaSource + Broker/Channel or a Broker/Channel + KafkaSink

[devguyio]

Problem

Knative Eventing API model does a good job addressing the basic needs for building an EDA:

• Sources for ingressing events.
• Channels and Brokers for events routing.
• Sinks for egressing events.

And that model was built in isolation initially which leads to limitations and shortcomings when combining multiple of those.

Advanced routing of eventing from an existing Kafka topic

A common use case for Knative Eventing is routing events in an existing Kafka topic beyond the simple point-to-point capabilities of the KafkaSource, for example content-based routing as with the Trigger's filters or topological routing as with the channels. This use case is already implemented at the API level by combining Source + Broker/Channel, however it comes with the following shortcomings:

1. The events get duplicated in the its trip from Kafka Topic-> Source -> Broker -> (Broker's Kafka Topic) -> Subscriber.
2. The sources have no Triggers Filters awareness. All events gets ingressed and duplicated to the Broker's topic before any filtering takes place regardless of existing Trigger filters.

With those shortcomings, Knative Eventing can be better optimized for the needs of some production use cases that require efficiency and cost-savings. (Some similar concerns were discussed in the past : knative/eventing#2460.

Feature

Avoid duplication of events in an existing Kafka topic when using any of these combinations:

• KafkaSource -> Broker/Channel
• Broker/Channel -> KafkaSink
• KafkaSource -> KafkaSink

Persona:

Which persona is this feature for?

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[pierDipi]

/remove-lifecycle stale /triage accepted

[xqianwang]

Any updates on this feature? I think is feature is really needed and it will make a lot improvements.

[pierDipi]

Hi @xqianwang, can you share your use case?