Is there any way to use Broker/Trigger without creating additional Kafka topics?

casssoft commented 3 years ago

Problem

In my cluster I already have a Kakfa topic managed through a non-knative system. I want a Broker/Trigger setup to read events off that kafka topic and filter them to different services, BUT ideally I don't want to have the overhead of an extra kafka topic just to copy the event into the Broker system.

This is a follow up from https://github.com/knative-sandbox/eventing-kafka/issues/215#issuecomment-733891085

The reason I want to use the Broker/Trigger setup is that it supports event filtering. Also it seems like the best way to use knative eventing when you have a big bucket of events that you want to separate out to multiple knative services

Here's the best ways I could tackle this that I know of:

Just deal with the overhead: Create a KafkaSource to send events to a Broker that's implemented with an MTChannelBasedBroker + KafkaChannel which would create a new kafka topic (Or Apache Kafka broker which I think would work the same way). The downside for this is that there would be an extra kafka topic and I assume extra latency, worse throughput, and more data usage.
Avoid using a Broker and create multiple KafkaSources with filtering to go to different knative services. (Thanks @pierDipi for the suggestion.) The downside is that this is a little bit harder to manage since each KafkaSource needs all the info to be able to connect to the kafka cluster (bootstrap servers/sasl/tls).
Try to force the KafkaChannel that underpins the broker to use the kafka topic that already exists. It seems like this is not the intended use case and currently has issues because knative will try to delete the topic (Related to https://github.com/knative-sandbox/eventing-kafka/issues/215)
Refactor the cluster so that Knative is in charge of the kafka topic instead of the external system and have the external system push either directly into the topic or into some type of source.
Create a new broker implementation to use existing kafka topics.

Persona: Event consumer

Exit Criteria A clear path for this use case.

(Sorry that this is vague, part of the reason for this issue is to move the discussion out of issue 215 and hopefully help guide others in a similar position.)

Additional context (optional) See the discussion ending with this comment: https://github.com/knative-sandbox/eventing-kafka/issues/215#issuecomment-733891085

@pierDipi Let me know if you had something else in mind when you said to create an issue.

Also I really appreciate the effort to guide me in the right direction!

lionelvillard commented 3 years ago

/cc @aslom

github-actions[bot] commented 3 years ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

lionelvillard commented 3 years ago

/remove-lifecycle stale

slinkydeveloper commented 3 years ago

I believe the eventing-kafka-broker can easily support this use case with a simple flag that just doesn't create a topic if it's not there... WDYT @pierDipi?

pierDipi commented 3 years ago

Yeah, we can support it. We just need to decide the API surface and then it could even be a good first contribution.

slinkydeveloper commented 3 years ago

@pierDipi can you open an issue? Whatever API is fine for me, even an annotation

github-actions[bot] commented 3 years ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

pierDipi commented 3 years ago

cc @itsmurugappan, this should be related to what you were sharing on #eventing-delivery regarding using a user-provided topic.

pierDipi commented 3 years ago

I opened an issue https://github.com/knative-sandbox/eventing-kafka-broker/issues/977

itsmurugappan commented 3 years ago

yes @pierDipi this is what kafka-topic-channel addresses and this is how I defined the channel spec -> https://github.com/Optum/kafka-topic-channel/blob/main/samples/pingsourceevents/channel-cm.yaml. Broker works the same.

github-actions[bot] commented 3 years ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

pierDipi commented 3 years ago

/remove-lifecycle stale

github-actions[bot] commented 2 years ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

pierDipi commented 2 years ago

/remove-lifecycle stale /triage accepted

knative-extensions / eventing-kafka

Is there any way to use Broker/Trigger without creating additional Kafka topics? #233