search

kyma-project / kyma

Kyma is an opinionated set of Kubernetes-based modular building blocks, including all necessary capabilities to develop and run enterprise-grade cloud-native applications.
https://kyma-project.io
Apache License 2.0
1.52k stars 405 forks source link

Triggers don't get ready when deleted/created at scale #10480

Closed sayanh closed 3 years ago

sayanh commented 3 years ago

Description

It is observed that when triggers(count=12) are deleted and created, they don't get ready.

The kafkachannels.knativekafka.kyma-project.io, default-kne-trigger did not pick up the new triggers.

The corresponding subscriptions were not ready as well.

The status for the subscription was:

status:
    conditions:
    - lastTransitionTime: "2021-01-28T10:41:24Z"
      status: "True"
      type: BrokerReady
    - lastTransitionTime: "2021-01-28T10:41:24Z"
      status: "True"
      type: DependencyReady
    - lastTransitionTime: "2021-01-28T10:41:24Z"
      message: 'Failed to get subscription status: [subscription "default-intelligent-knuth-2567d011-acc8-43f8-b162-635a690e333c"
        not present in channel "default-kne-trigger" subscriber''s list]'
      reason: SubscriptionNotMarkedReadyByChannel
      status: Unknown
      type: Ready
    - lastTransitionTime: "2021-01-28T10:41:24Z"
      message: 'Failed to get subscription status: [subscription "default-intelligent-knuth-2567d011-acc8-43f8-b162-635a690e333c"
        not present in channel "default-kne-trigger" subscriber''s list]'
      reason: SubscriptionNotMarkedReadyByChannel
      status: Unknown
      type: Subscribed
    - lastTransitionTime: "2021-01-28T10:41:24Z"
      status: "True"
      type: SubscriberResolved
    observedGeneration: 1
    subscriberURI: http://c4c-events-consumer.default.svc.cluster.local:80/

The status for triggers looked as follows:

status:
    conditions:
    - lastTransitionTime: "2021-01-28T10:41:23Z"
      status: "True"
      type: BrokerReady
    - lastTransitionTime: "2021-01-28T10:41:23Z"
      status: "True"
      type: DependencyReady
    - lastTransitionTime: "2021-01-28T10:41:23Z"
      message: 'Failed to get subscription status: [subscription "default-beautiful-mahavira-8be50c07-f6ff-4e71-b8da-d765c2db38a4"
        not present in channel "default-kne-trigger" subscriber''s list]'
      reason: SubscriptionNotMarkedReadyByChannel
      status: Unknown
      type: Ready
    - lastTransitionTime: "2021-01-28T10:41:23Z"
      message: 'Failed to get subscription status: [subscription "default-beautiful-mahavira-8be50c07-f6ff-4e71-b8da-d765c2db38a4"
        not present in channel "default-kne-trigger" subscriber''s list]'
      reason: SubscriptionNotMarkedReadyByChannel
      status: Unknown
      type: Subscribed
    - lastTransitionTime: "2021-01-28T10:41:23Z"
      status: "True"
      type: SubscriberResolved
    observedGeneration: 1
    subscriberURI: http://c4c-events-consumer.default.svc.cluster.local:80/

Logs from eventing-controller:

8T13:24:30.741Z","logger":"controller.subscription-controller.event-broadcaster","caller":"record/event.go:274","msg":"Event(v1.ObjectReference{Kind:\"Subscription\", Namespace:\"default\", Name:\"default-unruffled-bassi-bab9047f-3dda-421f-8916-a5fbc70694c2\", UID:\"ba26c4aa-ddf9-4297-977c-d49feb3b6732\", APIVersion:\"messaging.knative.dev/v1alpha1\", ResourceVersion:\"1037565\", FieldPath:\"\"}): type: 'Warning' reason: 'SubscriptionNotMarkedReadyByChannel' subscription \"default-unruffled-bassi-bab9047f-3dda-421f-8916-a5fbc70694c2\" not present in channel \"default-kne-trigger\" subscriber's list","commit":"8a92a8c","knative.dev/controller":"subscription-controller"}

Turns out the channel controller did not reconcile. The workaround was to delete the kafkachannel for filter in the user namespace to force reconciliation:

k delete kafkachannels.knativekafka.kyma-project.io default-kne-trigger
marcobebway commented 3 years ago

Not applicable.