Inconsistent "Unsubscribing from channel", unsubscribing a Subscription not in the channel

[maschmid]

Describe the bug kafka-ch-dispatcher logs show "Unsubscribing from channel" messages that shows subscriptions that belong to a different channel. This suggests the dispatcher possibly unsubscribes a different subscription that was meant to be unsubscribed.

cat kafka-ch-dispatcher-8664d75457-7sxh7.previous.log  | grep "Subscribing\|Unsubscribing"
{"level":"info","ts":"2021-01-26T15:00:33.125Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:340","msg":"Subscribing{channelRef 23 0  {kafka-stable-kn-retry-2 channel-0}} {subscription 23 0  190bb72f-5856-46a4-a66d-09b5a8a3465e}","knative.dev/pod":"kafka-ch-dispatcher-8664d75457-7sxh7"}
{"level":"info","ts":"2021-01-26T15:00:33.177Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:340","msg":"Subscribing{channelRef 23 0  {kc-broker-stable-dlq-2 receiver-dlq-0}} {subscription 23 0  5aa3779f-aee0-4d1c-8300-9af36b5c7694}","knative.dev/pod":"kafka-ch-dispatcher-8664d75457-7sxh7"}
{"level":"info","ts":"2021-01-26T15:00:33.254Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:340","msg":"Subscribing{channelRef 23 0  {kc-stable-kn-3 channel-0}} {subscription 23 0  3dfb5f6e-a3e2-40e9-ad6c-c4dee04b4b71}","knative.dev/pod":"kafka-ch-dispatcher-8664d75457-7sxh7"}
{"level":"info","ts":"2021-01-26T15:00:33.299Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:340","msg":"Subscribing{channelRef 23 0  {kc-newsubs-kn-1 channel-0}} {subscription 23 0  17b48839-a584-4d3e-8c67-20de848d7383}","knative.dev/pod":"kafka-ch-dispatcher-8664d75457-7sxh7"}
{"level":"info","ts":"2021-01-26T15:00:47.572Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:340","msg":"Subscribing{channelRef 23 0  {kc-broker-stable-dlq-2 broker-0-kne-trigger}} {subscription 23 0  37fe21a9-c642-4f22-b734-89a74f9997db}","knative.dev/pod":"kafka-ch-dispatcher-8664d75457-7sxh7"}
{"level":"info","ts":"2021-01-26T15:00:52.333Z","logger":"kafkachannel-dispatcher","caller":"dispatcher/dispatcher.go:373","msg":"Unsubscribing from channel","knative.dev/pod":"kafka-ch-dispatcher-8664d75457-7sxh7","channel":{"Namespace":"kafka-stable-kn-retry-2","Name":"channel-0"},"subscription":"UID: 37fe21a9-c642-4f22-b734-89a74f9997db\nSubscriber: http://broker-filter.knative-eventing.svc.cluster.local/triggers/kc-broker-stable-dlq-2/receiver-0/7b8ef69a-a188-40cc-bcbd-a1fb0dcd9f3e\nReply: http://broker-ingress.knative-eventing.svc.cluster.local/kc-broker-stable-dlq-2/broker-0\nDeadLetter: http://receiver-dlq-0-kn-channel.kc-broker-stable-dlq-2.svc.cluster.local\n"}

Note, the "Unsubscribing from channel" msg here, showing a Subscription 37fe21a9-c642-4f22-b734-89a74f9997db being unsubscribed from {"Namespace":"kafka-stable-kn-retry-2","Name":"channel-0"} , while it was originally subscribed to kc-broker-stable-dlq-2 broker-0-kne-trigger , which is a different channel in a different namespace.

Expected behavior No inconsistent logs. Dispatcher should not unsubscribe unrelated channels or subscriptions.

To Reproduce Steps to reproduce the behavior are unknown at this point, for now this is just something suspicious noticed in the logs. The test used variously configured brokers in several namespaces.

Knative release version eventing 0.19.2, eventing-kafka 0.19.1

Additional context

[maschmid]

An example logs with the occurence of this issue:

https://gist.github.com/maschmid/a20efbfaba9f076ebb02da1be13d26a9

[maschmid]

/assign @slinkydeveloper