New namespace, kong resource processing failed: failed to fetch the secret

[Niedman]

I've just created a new namespace in the cluster where I have other instances running kong and having ingresses working just fine. If i create a new ingress in any of the other namespaces it works but in the new namespace i get the error in kong logs:

ingress-controller time="2023-11-28T08:17:48Z" level=error msg="resource processing failed: failed to fetch the secret (sandbox/partner-sandbox-xxx-app)" GVK="networking.k8s.io/v1, Kind=Ingress" name=partner-api namespace=sandbox │

The ingress is defined in helm so it is pretty much the same with a different name.

The namespace and the google DNS zone where created new. Not really sure why it's not fetching the secret. The secret is create by the cert-manager

[rainest]

Does the Secret actually exist? I'm unsure whether you meant that cert-manager actually created it and you can kubectl get -n sandbox partner-sandbox-xxx-app successfully or if cert-manager is just configured to create it.

Which KIC version are you using, and does the issue persist if you make some trivial update to the Ingress (add a garbage annotation or similar). We start tracking Secrets based on their attachment to an Ingress, so I'm curious if there was maybe some race condition bug that failed to start tracking it--updating the Ingress forces a rescan of it and any attached resources.

Namespace-related issues shouldn't happen unless you've manually created RBAC configuration for your KIC service account and forgotten Secret read access in a namespace. There is also the --watch-namespaces feature, but that applies to all resources--if it was misconfigured, the controller would not see the Ingress either.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[codingwwws]

Hi @rainest I am using a cert for mtls inside a Service resource with:

  annotations:
    konghq.com/client-cert: my-service-secret
    konghq.com/protocol: https

Im getting the same error:

level=error msg="resource processing failed: failed to fetch secret 'my-service/my-service-secret': Secret my-service/my-service-secret not found" GVK="/v1, Kind=Service" name=my-service namespace=my-service

The Secret is there in the cluster and things are working fine. However once is deleted and recreated (renewed) by cert manager, the Ingress controller goes into the above error and the upstream service that is expecting the cert is receiving none. Steps to reproduce: 1- Setup a working mtla cert 2- Delete the secret containing the cert 3- cert manager will renew the mtls cert, recreating the Secret (same name, just new uid and resourceVersion and the cert content is changed) 4- The above error starts and that route to that Service doesn't send the new mtls cert to the upstream

Steps to solve currently is to add a dummy change to some annotation to the Service resource. Then as you suspected the issue goes away and the cert is reloaded correctly . Or to restart the Kong and kong ingress controller pods. That also loads teh new cert.

In effect the hot reload of this cert is broken. KIC 2.12.3 Kong 3.4.2