search

Kong / charts

Helm chart for Kong
Apache License 2.0
237 stars 473 forks source link

Kong unable to fetch JWT credentials #1055

Open Anupam5972 opened 2 months ago

Anupam5972 commented 2 months ago

For the Versions

Deployed Using helm Charts

The credentials are simultaneously created while the kong is deployed.

I am not facing this error for Version 2.10.5 but any version above that from 2.11.0 I am facing this issue


time="2024-03-25T14:04:51Z" level=error msg="resource processing failed: credential \"kong-jwt-credential\" failure: failed to fetch secret: Secret XXXXX/kong-jwt-credential not found" GVK="configuration.konghq.com/v1, Kind=KongConsumer" name=jwt-consumer namespace=XXXXX
time="2024-03-25T14:04:54Z" level=error msg="resource processing failed: credential \"kong-jwt-credential\" failure: failed to fetch secret: Secret XXXXX/kong-jwt-credential not found" GVK="configuration.konghq.com/v1, Kind=KongConsumer" name=jwt-consumer namespace=XXXXX.```
rainest commented 2 months ago

It looks like you upgraded from an older version, did you update your CRDs?

Though we don't have a full understanding of why it happens, we have previously seen this issue where the controller becomes unable to fetch resources when using an outdated set of CRDs.

Anupam5972 commented 2 months ago

I have my CRDs upto date, I rechecked it again and yeah it is

Anupam5972 commented 2 months ago

I'm using the controller-gen.kubebuilder.io/version: v0.13.0 I used Helm to deploy it chart version "kong-2.28.0" I want to Upgrade the KIC version from 2.10.5 to 2.12.3 but it's not fetching the credentials from 2.11.x itself

davidmontoyago commented 1 month ago

Hi everyone, our team is also experiencing this issue after upgrading from KIC 2.3 to 2.12.0 and 2.12.3. In our case it happens for all types of credentials. That is, JWT credentials, API keys and ACL groups. The ingress controller container just seems unable to read any k8s Secrets. We've verified the CRDs version and they match version 0.7.0 as seen below:

$ kubectl get crd kongconsumers.configuration.konghq.com -ojson | grep '"controller-gen.kubebuilder.io/version":'
"controller-gen.kubebuilder.io/version": "v0.7.0"

We've also verified that the Ingress controller's Service Account has enough permissions to fetch k8s Secrets in any namespace.

This is a long shot. Could this be related to the new way the Service Account token is now mounted with a projected volume? I'm thinking maybe the k8s client is not getting initialized with the proper SA token.

Cross-referencing https://github.com/Kong/kubernetes-ingress-controller/issues/5784 and https://github.com/Kong/kubernetes-ingress-controller/issues/5710 as they seem related.