search

knative-extensions / eventing-kafka

Kafka integrations with Knative Eventing.
Apache License 2.0
77 stars 82 forks source link

Immense memory usage for Kafka Cluster Receiver and KafkaChannel Pods #472

Closed FischlerA closed 3 years ago

FischlerA commented 3 years ago

Describe the bug We have created a MtChannelBasedBroker with a KafkaChannel and nothing else. The connection to our kafka cluster works just fine, we can read/write messages and so on. However after letting the system run on it's own without any kind of interaction and without deploying any resources besides the Broker (which also creates the KafkaChannel) the system suddenly starts using a lot of memory, to the point where our Kubernetes Worker Nodes run OutOfMemory and cause the pods to crash, forcing a restart and then the system starts working again until it all repeats.

We are using the KafkaChannel Distributed.

The Problem occurs with the KafkaClusterReceiver image

and the KafkaChannel image

KafkaChannel Logs

``` {"level":"info","ts":"2021-03-18T07:51:18.144Z","caller":"logging/config.go:116","msg":"Successfully created the logger."} {"level":"info","ts":"2021-03-18T07:51:18.145Z","caller":"logging/config.go:117","msg":"Logging level set to: info"} {"level":"info","ts":"2021-03-18T07:51:18.146Z","caller":"logging/config.go:79","msg":"Fetch GitHub commit ID from kodata failed","error":"open /var/run/ko/HEAD: no such file or directory"} {"level":"info","ts":"2021-03-18T07:51:18.164Z","logger":"eventing-kafka-channel-receiver","caller":"k8s/context.go:56","msg":"Setting Logging ConfigMap Watcher"} {"level":"info","ts":"2021-03-18T07:51:18.270Z","logger":"eventing-kafka-channel-receiver","caller":"env/environment.go:131","msg":"Environment Variables","Environment":{"SystemNamespace":"knative-eventing","MetricsPort":8081,"MetricsDomain":"eventing-kafka","PodName":"kafka-cluster-32603413-receiver-f9bb56885-dghx7","ContainerName":"kafkachannel-receiver","HealthPort":8082,"ServiceName":"kafka-cluster-32603413-receiver","ResyncPeriod":36000000000000,"KafkaSecretName":"kafka-cluster","KafkaSecretNamespace":"knative-eventing"}} {"level":"info","ts":"2021-03-18T07:51:18.293Z","logger":"eventing-kafka-channel-receiver","caller":"configmap/store.go:154","msg":"config-tracing-store config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}"} {"level":"info","ts":"2021-03-18T07:51:18.389Z","logger":"eventing-kafka-channel-receiver","caller":"profiling/server.go:64","msg":"Profiling enabled: false"} {"level":"info","ts":"2021-03-18T07:51:18.399Z","logger":"eventing-kafka-channel-receiver","caller":"metrics/exporter.go:161","msg":"Flushing the existing exporter before setting up the new exporter."} {"level":"info","ts":"2021-03-18T07:51:18.400Z","logger":"eventing-kafka-channel-receiver","caller":"metrics/prometheus_exporter.go:51","msg":"Created Prometheus exporter with config: &{eventing-kafka eventing_kafka prometheus 5000000000 false 8081 0.0.0.0 false { false}}. Start the server for Prometheus exporter."} {"level":"info","ts":"2021-03-18T07:51:18.400Z","logger":"eventing-kafka-channel-receiver","caller":"metrics/exporter.go:174","msg":"Successfully updated the metrics exporter; old config: ; new config &{eventing-kafka eventing_kafka prometheus 5000000000 false 8081 0.0.0.0 false { false}}"} {"level":"info","ts":"2021-03-18T07:51:18.496Z","logger":"eventing-kafka-channel-receiver","caller":"channel/channel.go:62","msg":"Successfully Initialized KafkaChannel Lister"} {"level":"info","ts":"2021-03-18T07:51:18.496Z","logger":"eventing-kafka-channel-receiver","caller":"health/health.go:102","msg":"Starting Server HTTP Server on port 8082"} {"level":"info","ts":"2021-03-18T07:51:18.610Z","logger":"eventing-kafka-channel-receiver","caller":"producer/producer.go:66","msg":"Creating Kafka SyncProducer"} {"level":"info","ts":"2021-03-18T07:51:18.909Z","logger":"eventing-kafka-channel-receiver","caller":"producer/producer.go:72","msg":"Successfully Created Kafka SyncProducer"} {"level":"info","ts":"2021-03-18T07:51:18.909Z","logger":"eventing-kafka-channel-receiver","caller":"producer/producer.go:95","msg":"Successfully Started Kafka Producer"} ```

KafkaChannel Logs

``` {"level":"info","ts":"2021-03-18T08:15:05.553Z","caller":"logging/config.go:116","msg":"Successfully created the logger."} {"level":"info","ts":"2021-03-18T08:15:05.553Z","caller":"logging/config.go:117","msg":"Logging level set to: info"} {"level":"info","ts":"2021-03-18T08:15:05.553Z","caller":"logging/config.go:79","msg":"Fetch GitHub commit ID from kodata failed","error":"open /var/run/ko/HEAD: no such file or directory"} {"level":"info","ts":"2021-03-18T08:15:05.561Z","logger":"eventing-kafka-channel-dispatcher","caller":"k8s/context.go:56","msg":"Setting Logging ConfigMap Watcher"} {"level":"info","ts":"2021-03-18T08:15:05.661Z","logger":"eventing-kafka-channel-dispatcher","caller":"env/environment.go:145","msg":"Environment Variables","Environment":{"SystemNamespace":"knative-eventing","MetricsPort":8081,"MetricsDomain":"eventing-kafka","PodName":"aftestbroker-1-kne-trigger-knative-eventing-6d3a1cfa-dispalk52w","ContainerName":"kafkachannel-dispatcher","HealthPort":8082,"KafkaTopic":"knative-eventing.aftestbroker-1-kne-trigger","ChannelKey":"knative-eventing/aftestbroker-1-kne-trigger","ServiceName":"aftestbroker-1-kne-trigger-knative-eventing-6d3a1cfa-dispatcher","ResyncPeriod":36000000000000,"KafkaSecretName":"kafka-cluster","KafkaSecretNamespace":"knative-eventing"}} {"level":"info","ts":"2021-03-18T08:15:05.679Z","logger":"eventing-kafka-channel-dispatcher","caller":"configmap/store.go:154","msg":"config-tracing-store config \"config-tracing\" config was added or updated: &config.Config{Backend:\"none\", ZipkinEndpoint:\"\", StackdriverProjectID:\"\", Debug:false, SampleRate:0.1}"} {"level":"info","ts":"2021-03-18T08:15:05.776Z","logger":"eventing-kafka-channel-dispatcher","caller":"profiling/server.go:64","msg":"Profiling enabled: false"} {"level":"info","ts":"2021-03-18T08:15:05.784Z","logger":"eventing-kafka-channel-dispatcher","caller":"metrics/exporter.go:161","msg":"Flushing the existing exporter before setting up the new exporter."} {"level":"info","ts":"2021-03-18T08:15:05.785Z","logger":"eventing-kafka-channel-dispatcher","caller":"metrics/prometheus_exporter.go:51","msg":"Created Prometheus exporter with config: &{eventing-kafka eventing_kafka prometheus 5000000000 false 8081 0.0.0.0 false { false}}. Start the server for Prometheus exporter."} {"level":"info","ts":"2021-03-18T08:15:05.785Z","logger":"eventing-kafka-channel-dispatcher","caller":"metrics/exporter.go:174","msg":"Successfully updated the metrics exporter; old config: ; new config &{eventing-kafka eventing_kafka prometheus 5000000000 false 8081 0.0.0.0 false { false}}"} {"level":"info","ts":"2021-03-18T08:15:05.881Z","logger":"eventing-kafka-channel-dispatcher","caller":"health/health.go:102","msg":"Starting Server HTTP Server on port 8082"} {"level":"info","ts":"2021-03-18T08:15:05.892Z","logger":"eventing-kafka-channel-dispatcher","caller":"dispatcher/dispatcher.go:324","msg":"No Consumer Changes Detected In New Configuration - Ignoring"} {"level":"info","ts":"2021-03-18T08:15:05.989Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:90","msg":"Setting Up Event Handlers"} {"level":"info","ts":"2021-03-18T08:15:05.990Z","logger":"eventing-kafka-channel-dispatcher","caller":"dispatcher/dispatcher.go:348","msg":"No relevant changes in Secret; ignoring update"} {"level":"info","ts":"2021-03-18T08:15:05.990Z","logger":"eventing-kafka-channel-dispatcher","caller":"dispatcher/main.go:179","msg":"Starting informers."} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"dispatcher/main.go:186","msg":"Registering dispatcher as alive and ready"} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"dispatcher/main.go:191","msg":"Starting controllers."} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:455","msg":"Starting controller and workers"} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:465","msg":"Started workers"} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/default-broker-kne-trigger"} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000041835,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-samples/default-kne-trigger"} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:530","msg":"Reconcile succeeded","knative.dev/traceid":"6918d195-8392-45e7-8263-1e95888b813c","knative.dev/key":"knative-samples/default-kne-trigger","duration":0.000023201} {"level":"info","ts":"2021-03-18T08:15:06.091Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:530","msg":"Reconcile succeeded","knative.dev/traceid":"8b93defe-76c2-462e-8362-7ec09a671349","knative.dev/key":"knative-eventing/default-broker-kne-trigger","duration":0.000088271} {"level":"info","ts":"2021-03-18T08:15:06.096Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.096Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000074732,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:06.107Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.107Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000118979,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:06.127Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.128Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.00015924,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:06.168Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.168Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000075876,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:06.248Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.248Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000138303,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:06.409Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.409Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000151752,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:06.729Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:06.729Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000076032,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:07.370Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:07.370Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000112111,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:08.650Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:08.650Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000086873,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:11.211Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:11.211Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000114041,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:16.331Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:16.331Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000163847,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:26.572Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"error","ts":"2021-03-18T08:15:26.572Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:540","msg":"Reconcile error","duration":0.000092281,"error":"channel is not ready - cannot configure and update subscriber status","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:540\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:523\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20210216013737-584933f8280b/controller/controller.go:460"} {"level":"info","ts":"2021-03-18T08:15:39.895Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"info","ts":"2021-03-18T08:15:41.031Z","logger":"eventing-kafka-channel-dispatcher","caller":"dispatcher/dispatcher.go:195","msg":"ConsumerGroup Error Processing Initiated","GroupId":"kafka.e3882ccf-0d1d-46fc-ba72-2cb374a4c679"} {"level":"info","ts":"2021-03-18T08:15:41.031Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:162","msg":"Successfully Verified / Updated KafkaChannel Status"} {"level":"info","ts":"2021-03-18T08:15:41.031Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:530","msg":"Reconcile succeeded","knative.dev/traceid":"d3410693-75dc-44fe-8974-575a5989139a","knative.dev/key":"knative-eventing/aftestbroker-1-kne-trigger","duration":1.135720888} {"level":"info","ts":"2021-03-18T08:15:41.031Z","logger":"eventing-kafka-channel-dispatcher","caller":"dispatcher/dispatcher.go:220","msg":"ConsumerGroup Message Consumption Initiated","GroupId":"kafka.e3882ccf-0d1d-46fc-ba72-2cb374a4c679"} {"level":"info","ts":"2021-03-18T08:15:41.031Z","logger":"eventing-kafka-channel-dispatcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"KafkaChannel\", Namespace:\"knative-eventing\", Name:\"aftestbroker-1-kne-trigger\", UID:\"3e0cfbf3-92ec-4817-a065-6c436ee3a3b8\", APIVersion:\"messaging.knative.dev/v1beta1\", ResourceVersion:\"184441\", FieldPath:\"\"}): type: 'Normal' reason: 'ChannelReconciled' KafkaChannel Reconciled"} {"level":"info","ts":"2021-03-18T08:15:47.052Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/aftestbroker-1-kne-trigger"} {"level":"info","ts":"2021-03-18T08:15:47.052Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:162","msg":"Successfully Verified / Updated KafkaChannel Status"} {"level":"info","ts":"2021-03-18T08:15:47.052Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:530","msg":"Reconcile succeeded","knative.dev/traceid":"21e64c5a-fb8d-44f1-b818-999c0680bd88","knative.dev/key":"knative-eventing/aftestbroker-1-kne-trigger","duration":0.000231465} {"level":"info","ts":"2021-03-18T08:15:47.053Z","logger":"eventing-kafka-channel-dispatcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"KafkaChannel\", Namespace:\"knative-eventing\", Name:\"aftestbroker-1-kne-trigger\", UID:\"3e0cfbf3-92ec-4817-a065-6c436ee3a3b8\", APIVersion:\"messaging.knative.dev/v1beta1\", ResourceVersion:\"184441\", FieldPath:\"\"}): type: 'Normal' reason: 'ChannelReconciled' KafkaChannel Reconciled"} {"level":"info","ts":"2021-03-18T08:38:57.364Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/default-broker-kne-trigger"} {"level":"info","ts":"2021-03-18T08:38:57.364Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:530","msg":"Reconcile succeeded","knative.dev/traceid":"91476ef1-c355-4c3b-8e50-952e2df7da35","knative.dev/key":"knative-eventing/default-broker-kne-trigger","duration":0.008801098} {"level":"info","ts":"2021-03-18T08:39:30.447Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/kafkachannel.go:115","msg":"Reconcile","key":"knative-eventing/default-broker-kne-trigger"} {"level":"info","ts":"2021-03-18T08:39:30.448Z","logger":"eventing-kafka-channel-dispatcher","caller":"controller/controller.go:530","msg":"Reconcile succeeded","knative.dev/traceid":"4d8f5253-487b-42ed-a7e0-a6452accd204","knative.dev/key":"knative-eventing/default-broker-kne-trigger","duration":0.000500178} ```

Expected behavior The memory usage should not suddenly increase.

To Reproduce Create a MtChannelBased Broker which uses KafkaChannel.

Knative release version v0.21.0

Additional context None

If you need any more information please let me know.

slinkydeveloper commented 3 years ago

In order to investigate this issue, we need either an indication of your load behaviour and some profiling data from pprof

FischlerA commented 3 years ago

In order to investigate this issue, we need either an indication of your load behaviour and some profiling data from pprof

That is exactly the issue. There is no load, we are neither sending nor receiving any data. The system is just idle. Currently it's just a proof of concept, so we have empty kafka topics and without any kind of interaction (i was literally in the kitchen drinking coffee) the memory usage spikes until the node ran out of memory and the cycle repeated.

It also happens at irregular intervals. (the following image uses different colors because the pods keep restarting after the spike) image

After still having the issue we tried deploying another MtChannelBasedBroker with a backing Kafka Channel in another namespace but the same thing happened.

slinkydeveloper commented 3 years ago

@FischlerA Does this happen using deploying directly some KafkaChannel, keeping them idle, without deploying any broker?

FischlerA commented 3 years ago

@FischlerA Does this happen using deploying directly some KafkaChannel, keeping them idle, without deploying any broker?

It happens when deploying a Broker that creates the KafkaChannel for us and keeping both of them idle. here are the configmaps used to create broker and channel

config-br-defaults configmap

``` default-br-config: | clusterDefault: brokerClass: MTChannelBasedBroker apiVersion: v1 kind: ConfigMap name: config-br-default-channel namespace: knative-eventing ```

config-br-default-channel

``` channelTemplateSpec: |- apiVersion: messaging.knative.dev/v1beta1 kind: KafkaChannel spec: numPartitions: 3 replicationFactor: 3 ```

And here is the yaml files for creating the broker

apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
  name: aftestbroker-1
  namespace: knative-eventing
slinkydeveloper commented 3 years ago

This seems like a normal setup... I'm sorry but I can't help any further without some indications on where the problem is... Maybe do you have some metrics on the topics activity? To see if, for some reason, there are messages going around even if idle? Or some more granular indication of which thing is draining so much memory. A profiling of the channel dispatcher pod would be extremely helpful, because it can give us an hint of where the issue is.

The fact that the kafka cluster is using so much memory is also weird, maybe some misconfiguration of kafka?

FischlerA commented 3 years ago

@slinkydeveloper i am really sorry to inform you that my customer deceided that he doesn't want us to spend anymore time on knative. I am really, truly sorry, but i won't be able to provide anymore information without going directly against the wishes of my customer which i just can't do.

The last part of information i can provide you is that we are using confluent cloud to host our kafka.

I am really sorry but i have to close this issue.