search

knative-extensions / eventing-kafka-broker

Alternate Kafka Broker implementation.
Apache License 2.0
170 stars 116 forks source link

KafkaNamespaced for KafkaSources #4118

Open Zurvarian opened 2 hours ago

Zurvarian commented 2 hours ago

Problem Right now we have different namespaces that represents different usages and each of them could require different needs when it comes to configuring the Kafka consumer.

Persona: Which persona is this feature for? System Operator

Exit Criteria I would like to be able to configure my KafkaSource CRDs to allow the control plane to create independent Data Planes per namespace where we could have overrides in the Kafka Consumer config. Ideally I should also be able to configure the Mem/CPU requests and limits of these other instances.

Time Estimate (optional): How many developer-days do you think this may take to resolve? I can't say, but given the source code is shared between Broker and KafkaSource, the implementation should not be that costly.

Additional context (optional) Nothing else to add.

pierDipi commented 2 hours ago

@Zurvarian would a different solution outlined here https://github.com/knative-extensions/eventing-kafka-broker/issues/1412 work for you? It would still keep the shared data plane but allow to configure more consumer and producer configurations

Zurvarian commented 2 hours ago

It would for the time being, but at some point later we plan to have independent NodePools per client and then we will want to have different StatefulSets using the resources of said NodePools only. So we can isolate resource consumption in-between clients.

pierDipi commented 47 minutes ago

Can you describe in more details why is that level of isolation in your use case necessary ?

Zurvarian commented 37 minutes ago

Can't give much details because of the clients list is private, but the overall idea is that the clients could be charged based on how much resources they use in our system, so the final goal is to split the CPU/Memory usage across different NodePools so the K8S resources used per client is easily identifiable.

A valid alternative for us would be to deploy independent Knative Eventing deployments in each of the NodePools / namespaces, but then the control plane resources being shared is not an issue and it will just require more maintenance to keep all versions aligned. Though, AFAIK, it is not possible to restrict which KafkaSource definitions are handled by the Controller, isn't it? Something like a list of Namespaces to look for would do.