search

knative / eventing

Event-driven application platform for Kubernetes
https://knative.dev/docs/eventing
Apache License 2.0
1.42k stars 596 forks source link

Channel's subscription don't update when the trigger deleted. #6356

Closed zhaojinxin409 closed 1 year ago

zhaojinxin409 commented 2 years ago

Describe the bug I created broker and some trigger subscribing the broker, the kafkachannel's config is correct. Later when i delete the trigger the channel's subscription according to the trigger still exists in the channel's subcriptions.

Knative release version Using the kafkadistributed channel v1.1

Logs detailed log of eventing-controller: https://gist.github.com/zhaojinxin409/07db877742a8d637faa573a860c590c1

{"level":"warn","ts":"2022-04-26T09:49:43.643Z","logger":"controller","caller":"subscription/subscription.go:136","msg":"Failed to get subscription status.","commit":"b3bccaf","knative.dev/pod":"eventing-controller-6797496b59-dsjnf","knative.dev/controller":"knative.dev.eventing.pkg.reconciler.subscription.Reconciler","knative.dev/kind":"messaging.knative.dev.Subscription","knative.dev/traceid":"680e2bde-746c-49d9-99b2-cd04776116d4","knative.dev/key":"cosmoim-integration-prod/e-f4ef66c69088d108-hmqm-integra661895d143221602b74487e1199bcd9c","error":"subscription \"e-f4ef66c69088d108-hmqm-integra661895d143221602b74487e1199bcd9c\" not present in channel \"e-f4ef66c69088d108-kne-trigger\" subscriber's list"}
{"level":"error","ts":"2022-04-27T02:28:20.634Z","logger":"controller","caller":"controller/controller.go:566","msg":"Reconcile error","commit":"b3bccaf","knative.dev/pod":"eventing-controller-6797496b59-dsjnf","knative.dev/controller":"knative.dev.eventing.pkg.reconciler.subscription.Reconciler","knative.dev/kind":"messaging.knative.dev.Subscription","knative.dev/traceid":"fa9d6f94-7e4c-4dd1-94f4-a4a373b5845c","knative.dev/key":"cosmoim-integration-prod/e-f4ef66c69088d108-hmqm-integra8fb8adf3412270f718703c8eae2de9a3","duration":0.09527195,"error":"failed to clear finalizers: subscriptions.messaging.knative.dev \"e-f4ef66c69088d108-hmqm-integra8fb8adf3412270f718703c8eae2de9a3\" not found","stacktrace":"knative.dev/pkg/controller.(*Impl).handleErr\n\tknative.dev/pkg@v0.0.0-20211206113427-18589ac7627e/controller/controller.go:566\nknative.dev/pkg/controller.(*Impl).processNextWorkItem\n\tknative.dev/pkg@v0.0.0-20211206113427-18589ac7627e/controller/controller.go:543\nknative.dev/pkg/controller.(*Impl).RunContext.func3\n\tknative.dev/pkg@v0.0.0-20211206113427-18589ac7627e/controller/controller.go:478"}

Additional context Trigger: no hmqm-integration-prod-hmqm-test-1 exists

$ kubectl -n cosmoim-integration-prod get trigger hmqm-integration-prod-hmqm-test-1
Error from server (NotFound): triggers.eventing.knative.dev "hmqm-integration-prod-hmqm-test-1" not found

Subscription: no hmqm-integration**** exists

Channel's config, notice there are 3 subscriptions hmqm-integration-prod-hmqm-test-1 in namespace cosmoim-integration-prod

apiVersion: messaging.knative.dev/v1beta1
kind: KafkaChannel
metadata:
  annotations:
    eventing.knative.dev/scope: cluster
    messaging.knative.dev/subscribable: v1
  creationTimestamp: "2022-03-22T01:50:21Z"
  finalizers:
  - kafkachannels.messaging.knative.dev
  generation: 101
  labels:
    eventing.knative.dev/broker: e-f4ef66c69088d108
    eventing.knative.dev/brokerEverything: "true"
    kafkaTopic: cosmoim-integration-prod.e-f4ef66c69088d108-kne-trigger
  name: e-f4ef66c69088d108-kne-trigger
  namespace: cosmoim-integration-prod
  ownerReferences:
  - apiVersion: eventing.knative.dev/v1
    blockOwnerDeletion: true
    controller: true
    kind: Broker
    name: e-f4ef66c69088d108
    uid: fc3b5f40-c869-4fcd-9f84-40204b268311
  resourceVersion: "498963381"
  uid: d96a04a7-7bf7-43ca-84e8-8ab84e0f7e2a
spec:
  numPartitions: 6
  replicationFactor: 1
  retentionDuration: PT168H
  subscribers:
  - generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-dock-1/d12275cc-d19c-4777-8833-6f9effd1b20e
    uid: 43bf86dc-56de-4d2c-b480-92a4528d1c75
  - generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-process-0/56482a59-4a08-4b19-84a5-61280c8d5c70
    uid: c05d2be2-6e7f-4165-9351-63d0f179da13
  - generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-dock-0/059758c4-4ea7-418b-9f8e-9f1880d5c0ef
    uid: a88b17e9-bb52-44f6-a97b-d5bdf62f7713
  - generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/gmdc-integration-prod-test-cosmo-mom-gmdc-driverinfo-0/2f4128ec-9e93-4f70-8c20-9632164fae90
    uid: de7a3e00-a3ea-41e7-870c-b6de3fc9c7f9
  - generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/e-f4ef66c69088d108/041f61ff-71e7-4ed5-9d9a-e19b3b31f498
    uid: cfc01295-c4e8-4b6c-acdb-b1f70d149a02
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fcosmoim-integration-prod&itId=test-cosmo-mom-de-intern-dock&itProject=cosmoim&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-dock-0/28aad4fe-732e-424f-8f34-d95d420b0aa9
    uid: 41dd7905-1ef5-4b11-ad55-c4d08951183b
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fcosmoim-integration-prod&itId=test-cosmo-mom-de-intern-dock&itProject=cosmoim&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-dock-1/cbf2d91c-4df5-4629-afaa-72639c1638c5
    uid: 5647696a-4284-436a-b7e8-7b0b3eb7689e
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fcosmoim-integration-prod&itId=test-cosmo-mom-de-intern-process&itProject=cosmoim&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-process-0/18cc59a6-0df9-42e8-9008-69657e40eb44
    uid: 261857e8-3263-41db-9313-19a22865a535
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fcosmoim-integration-prod&itId=test-cosmo-mom-de-intern-process&itProject=cosmoim&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-process-1/591fa026-e759-4e66-a8cd-669908e07975
    uid: 03480bc7-0b2c-4a81-aeac-e58658031d30
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fcosmoim-integration-prod&itId=test-cosmo-mom-de-intern-task&itProject=cosmoim&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/cosmoim-integration-prod-test-cosmo-mom-de-intern-task-0/7265b9b2-b02f-4a2b-ab63-b4c05dd7a643
    uid: c524ccff-afdf-487c-a550-2c5212431265
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhmqm-integration-prod&itId=hmqm-test&itProject=hmqm&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hmqm-integration-prod-hmqm-test-0/9af07add-4460-4a5b-b6af-1a2e6e7ce3bf
    uid: 7e61f1a3-bf03-47cd-892d-fc1942e90b9e
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhgwms-integration-prod&itId=hgwms-wmsrest-raw-out-test-mom&itProject=hgwms&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hgwms-integration-prod-hgwms-wmsrest-raw-out-test-mom-0/26b01b88-ce14-4e08-8e12-180cb21f4ba3
    uid: 9d27a9c9-bc86-43f5-bd8a-ada2c0c04f07
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhgwms-integration-prod&itId=hgwms-wmsrest-raw-out-test-mom&itProject=hgwms&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hgwms-integration-prod-hgwms-wmsrest-raw-out-test-mom-1/f132d25d-3ddc-4348-85d8-f3d03b7d3233
    uid: 724c2bda-fc29-482b-860d-34c166f31921
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhgwms-integration-prod&itId=hgwms-wmsrest-raw-out-test-mom&itProject=hgwms&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hgwms-integration-prod-hgwms-wmsrest-raw-out-test-mom-0/743150b9-0b67-4431-a81d-5e94157d7dbb
    uid: feeab6b8-17f8-4f61-9f4e-179f76f39c3c
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhmqm-integration-prod&itId=hmqm-dev&itProject=hmqm&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hmqm-integration-prod-hmqm-dev-0/31bdc095-6a46-42a6-ac0a-57fb67591287
    uid: 0049f2dd-5c8f-401d-84d6-25dcaf85d4ee
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhmqm-integration-prod&itId=hmqm-test&itProject=hmqm&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hmqm-integration-prod-hmqm-test-1/b944cb9a-ffcf-4692-acf4-283023da1d03
    uid: 66d8b319-c04f-4029-ae0d-04b5988de86b
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhmqm-integration-prod&itId=hmqm-test&itProject=hmqm&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hmqm-integration-prod-hmqm-test-1/33bd2229-732b-449b-8e66-4d637ae92293
    uid: a19d3885-cfa2-4203-9748-1c06e4b56eec
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhmqm-integration-prod&itId=hmqm-test&itProject=hmqm&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hmqm-integration-prod-hmqm-test-1/2dfdfaee-478a-46b6-913d-269bf2a37ef7
    uid: 2a1a37f0-816e-4c80-beaf-eed9532dfbc4
  - delivery:
      backoffDelay: PT1S
      backoffPolicy: exponential
      deadLetterSink:
        uri: http://sg.qd-aliyun-dmz-ack-internal.haier.net/event/trace?action=pushEvent&busId=e-f4ef66c69088d108&busProject=cosmoim&itCluster=qd-aliyun-dmz-ack-prod%2Fhmqm-integration-prod&itId=zjx-test&itProject=hmqm&status=failed
      retry: 3
    generation: 1
    replyUri: http://broker-ingress.knative-eventing.svc.cluster.local/cosmoim-integration-prod/e-f4ef66c69088d108
    subscriberUri: http://broker-filter.knative-eventing.svc.cluster.local/triggers/cosmoim-integration-prod/hmqm-integration-prod-zjx-test-0/1cfb6d92-250b-40ee-8e0c-9fec9acb57ca
    uid: 9cfc6a4f-d458-43c1-b1c4-d97e35bd548c
status:
  address:
    url: http://e-f4ef66c69088d108-kne-trigger-kn-channel.cosmoim-integration-prod.svc.cluster.local
  conditions:
  - lastTransitionTime: "2022-03-22T01:50:22Z"
    status: "True"
    type: Addressable
  - lastTransitionTime: "2022-03-22T01:50:22Z"
    status: "True"
    type: ChannelServiceReady
  - lastTransitionTime: "2022-03-22T01:50:22Z"
    status: "True"
    type: ConfigurationReady
  - lastTransitionTime: "2022-04-27T02:23:01Z"
    status: "True"
    type: DispatcherReady
  - lastTransitionTime: "2022-03-22T01:50:23Z"
    status: "True"
    type: EndpointsReady
  - lastTransitionTime: "2022-04-27T02:23:01Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2022-03-22T01:50:23Z"
    status: "True"
    type: ServiceReady
  - lastTransitionTime: "2022-03-22T01:50:22Z"
    status: "True"
    type: TopicReady
  observedGeneration: 101
  subscribers:
  - observedGeneration: 1
    ready: "True"
    uid: 43bf86dc-56de-4d2c-b480-92a4528d1c75
  - observedGeneration: 1
    ready: "True"
    uid: c05d2be2-6e7f-4165-9351-63d0f179da13
  - observedGeneration: 1
    ready: "True"
    uid: a88b17e9-bb52-44f6-a97b-d5bdf62f7713
  - observedGeneration: 1
    ready: "True"
    uid: de7a3e00-a3ea-41e7-870c-b6de3fc9c7f9
  - observedGeneration: 1
    ready: "True"
    uid: cfc01295-c4e8-4b6c-acdb-b1f70d149a02
  - observedGeneration: 1
    ready: "True"
    uid: 41dd7905-1ef5-4b11-ad55-c4d08951183b
  - observedGeneration: 1
    ready: "True"
    uid: 5647696a-4284-436a-b7e8-7b0b3eb7689e
  - observedGeneration: 1
    ready: "True"
    uid: 261857e8-3263-41db-9313-19a22865a535
  - observedGeneration: 1
    ready: "True"
    uid: 03480bc7-0b2c-4a81-aeac-e58658031d30
  - observedGeneration: 1
    ready: "True"
    uid: c524ccff-afdf-487c-a550-2c5212431265
  - observedGeneration: 1
    ready: "True"
    uid: 7e61f1a3-bf03-47cd-892d-fc1942e90b9e
  - observedGeneration: 1
    ready: "True"
    uid: 9d27a9c9-bc86-43f5-bd8a-ada2c0c04f07
  - observedGeneration: 1
    ready: "True"
    uid: 724c2bda-fc29-482b-860d-34c166f31921
  - observedGeneration: 1
    ready: "True"
    uid: feeab6b8-17f8-4f61-9f4e-179f76f39c3c
  - observedGeneration: 1
    ready: "True"
    uid: 0049f2dd-5c8f-401d-84d6-25dcaf85d4ee
  - observedGeneration: 1
    ready: "True"
    uid: 66d8b319-c04f-4029-ae0d-04b5988de86b
  - observedGeneration: 1
    ready: "True"
    uid: a19d3885-cfa2-4203-9748-1c06e4b56eec
  - observedGeneration: 1
    ready: "True"
    uid: 2a1a37f0-816e-4c80-beaf-eed9532dfbc4
  - observedGeneration: 1
    ready: "True"
    uid: 9cfc6a4f-d458-43c1-b1c4-d97e35bd548c
zhaojinxin409 commented 2 years ago

Updating:

I tried to restart the eventing-controller and the problem still exists. Then i delete the subscriber that can not be found in channel's subscribers list, the problem solved.

l-qing commented 1 year ago

Might be the same issue as this one: https://github.com/knative/eventing/issues/6636

Deletion of subscriptions occurs orphaned subscriber in channel implementaiton's spec

l-qing commented 1 year ago

Updating:

I tried to restart the eventing-controller and the problem still exists. Then i delete the subscriber that can not be found in channel's subscribers list, the problem solved.

You can try to delete the channel and wait for it to be rebuilt automatically.😁

zhaojinxin409 commented 1 year ago

It's the same problem, thanks😊