cosumers conflict between kafkasource and kafka-source-dispatcher

[ninoamine]

Describe the bug When I create kafkasource that consume messages from a topic with one partition, it creates a consumer groupe with 3 members one for the kafkasource and two for the kafka-source-dispatcher, so when I produce a message in my topic it is consumed by one of the consumer of kafka-source-dispatcher so it is kind of lost.

Expected behavior Normalement it should be one consumer of the kafkasource pod

knative-eventing version: 1.3.2

Additional context I don't understand why the dispatcher add those consumer it brokes the workflow

[pierDipi]

@ninoamine please provide steps to reproduce on a more recent version of Knative, 1.3.2 is quite old and the community doesn't support it anymore

[ninoamine]

@pierDipi thank you for the response, we are kind of stuck un this version because of some infrastrucutre dependencies, but can I at least know what is the use of the kafka-source-dispatcher and why does it need to create consumer beside the consume of the kafkasource pord ?

[pierDipi]

You need to run the post-install jobs to upgrade components properly

[pierDipi]

without such jobs you will get competing components trying to handle a single KafkaSource

[ninoamine]

we used the operator to install knative eventing, the post install is not managed by default in it ?

[pierDipi]

The operator would work if you install only KafkaSource but probably you are installing Knative Broker with class Kafka or KafkaChannel as well, in that case, the Knative Kafka components should not be managed by the operator (aka you should disable operator support for Kafka source)

[pierDipi]

Can you perhaps share more about your configurations, what does the KnativeEventing CR look like? what's your cluster state currently (ie list pods in knative-eventing and user's namespaces where KafkaSource are, etc)?

[ninoamine]

of course: this is the knativeeventing cr manifest kind: KnativeEventing metadata: name: knative-eventing namespace: knative-eventing spec: config: logging: loglevel.controller: debug loglevel.webhook: debug zap-logger-config: "{\n \"level\": \"debug\",\n \"development\": false,\n \"outputPaths\": [\"stdout\"],\n \"errorOutputPaths\": [\"stderr\"],\n \"encoding\": \"json\",\n \"encoderConfig\": {\n \"timeKey\": \"ts\",\n \"levelKey\": \"level\",\n \ \"nameKey\": \"logger\",\n \"callerKey\": \"caller\",\n \"messageKey\": \"msg\",\n \"stacktraceKey\": \"stacktrace\",\n \"lineEnding\": \"\",\n \ \"levelEncoder\": \"\",\n \"timeEncoder\": \"iso8601\",\n \"durationEncoder\": \"\",\n \"callerEncoder\": \"\"\n } \n}\n" high-availability: replicas: 1 source: kafka: enabled: true

we also installed : apache kafka controller https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.2/eventing-kafka-controller.yaml

kafka channel data plane https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.2/eventing-kafka-channel.yaml

kadka source data plane https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.2/eventing-kafka-source.yaml

apache kafka broker https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.2/eventing-kafka-broker.yaml

Apache Kafka Sink https://github.com/knative-sandbox/eventing-kafka-broker/releases/download/knative-v1.3.2/eventing-kafka-sink.yaml

and this the element that we have in knative-eventing ns NAME READY UP-TO-DATE AVAILABLE AGE eventing-controller 1/1 1 1 14d eventing-webhook 1/1 1 1 14d imc-controller 1/1 1 1 14d imc-dispatcher 1/1 1 1 14d kafka-broker-dispatcher 2/2 2 2 14d kafka-broker-receiver 1/1 1 1 14d kafka-channel-dispatcher 1/1 1 1 14d kafka-channel-receiver 1/1 1 1 14d kafka-controller 1/1 1 1 14d kafka-controller-manager 1/1 1 1 14d kafka-sink-receiver 1/1 1 1 14d kafka-source-dispatcher 0/0 0 0 14d kafka-webhook-eventing 1/1 1 1 14d mt-broker-controller 1/1 1 1 14d mt-broker-filter 1/1 1 1 14d mt-broker-ingress 1/1 1 1 14d pingsource-mt-adapter 0/0 0 0 14d sugar-controller 1/1 1 1 14d

[ninoamine]

maybe as you said we have some comflict with resources created by the operator and the one that we installed by manifest

[pierDipi]

Yes, that's the reason, you should disable kafka source in the KnativeEventing CR and re-apply eventing-kafka-broker repositories YAMLs (including the post-install artifact)

[ninoamine]

that was it, thank you @pierDipi a lot. Just an other question please for the option kafkanamespaced brokers in which version was it released ?

[pierDipi]

Just an other question please for the option kafkanamespaced brokers in which version was it released ?

1.8

[pierDipi]

Closing this as comments were addressed