failed fetching KongUpstreamPolicy v3.2.3

balait4 commented 4 weeks ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Upgraded Kong from v3.5.0 to v.3.7.1 and KIC from 3.0.2 to 3.2.3. Before upgrade we are able to use KongUpstreamPolicy, after upgrade we see the below error for all KongUpstreamPolicy. We have db-less setup and performed the manual upgrade by update the crd's and then we updated the new image in deployment.

kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/crd?ref=v3.2.3 | kubectl apply -f - kubectl kustomize https://github.com/Kong/kubernetes-ingress-controller/config/rbac?ref=v3.2.3 | kubectl apply -f -

 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "northamerica-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "britishisles-least-conn", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-least-conn not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "netherlands-funda-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}
 2024-09-06T13:13:35Z error recording a Warning event for object {"name": "westcentraleurope-lb-header", "namespace": "api-proxy", "kind": "Service", "apiVersion": "/v1", "reason": "KongConfigurationTranslationFailed", "message": "failed fetching KongUpstreamPolicy: KongUpstreamPolicy api-proxy/kongingress-lb-header not found", "error": "object failed to apply"}

Expected Behavior

Should see the KongUpstreamPolicy

Steps To Reproduce

DB-less setup upgrade kong from v3.5.0 to v.3.7.1 and KIC from 3.0.2 to 3.2.3.

Kong Ingress Controller version

v3.2.3

Kubernetes version

v1.28.2

Anything else?

No response

balait4 commented 1 week ago

Any help on this please?

gcamus59 commented 1 week ago

We face the same issue after upgrading to Kong 3.7.1 and KIC 3.2.3. Around 10 Kubernetes services referenced a KongUpstreamPolicy to add a LB strategy with hashOn IP. In the Kubernetes services events we have the following errors:

Events:
  Type     Reason                              Age                    From         Message
  ----     ------                              ----                   ----         -------
  Warning  KongConfigurationTranslationFailed  54m (x3 over 54m)      kong-client  failed fetching KongUpstreamPolicy: KongUpstreamPolicy apps-sbx/hashon-ip not found
  Warning  KongConfigurationTranslationFailed  53m (x4 over 53m)      kong-client  failed fetching KongUpstreamPolicy: KongUpstreamPolicy apps-sbx/hashon-ip not found
  Warning  KongConfigurationTranslationFailed  3m50s (x4 over 3m59s)  kong-client  failed fetching KongUpstreamPolicy: KongUpstreamPolicy apps-sbx/hashon-ip not found

The KongUpstreamPolicy is well created and available in the same namespace. We noticed that the KongUpstreamPolicy resource status block is continuously updated with ancestor reordering.

...
Status:
  Ancestors:
    Ancestor Ref:
      Group:      core
      Kind:       Service
      Name:       my-api-report
      Namespace:  apps-sbx
    Conditions:
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
    Controller Name:         konghq.com/kic-gateway-controller
    Ancestor Ref:
      Group:      core
      Kind:       Service
      Name:       my-api-swagger
      Namespace:  apps-sbx
    Conditions:
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-09-26T09:54:10Z
      Message:
      Reason:                Programmed
      Status:                True
      Type:                  Programmed
    Controller Name:         konghq.com/kic-gateway-controller

Kong / kubernetes-ingress-controller