Kong / charts

Helm chart for Kong
Apache License 2.0
247 stars 477 forks source link

Ingress Controller fails after update to chart version 2.37.1 #1083

Closed sebboer closed 3 months ago

sebboer commented 3 months ago

Hello, I have a few problems updating the kong helmet chart and I don't know where to start. I have updated from 2.36.0 to 2.37.1 and updated the crds as follows.

kubectl apply -f https://raw.githubusercontent.com/Kong/charts/kong-2.37.1/charts/kong/crds/custom-resource-definitions.yaml

The service account seems to have problems with the crds. KongLicenses cannot be listed and all other crds fail.

Here are some relevant logs of the failing ingress controller

2024-06-11 12:33:21.980 2024-06-11T10:33:21Z    info    Diagnostics server disabled {"v": 0}
2024-06-11 12:33:21.980 2024-06-11T10:33:21Z    info    setup   Starting controller manager {"v": 0, "release": "3.1.5", "repo": "https://github.com/Kong/kubernetes-ingress-controller.git", "commit": "d3e401dcb86ecc40a828eaf5da20e6c90a5bf323"}
2024-06-11 12:33:21.980 2024-06-11T10:33:21Z    info    setup   The ingress class name has been set {"v": 0, "value": "my-company-kong"}
2024-06-11 12:33:21.980 2024-06-11T10:33:21Z    info    setup   Getting enabled options and features    {"v": 0}
2024-06-11 12:33:21.980 2024-06-11T10:33:21Z    info    setup   Getting the kubernetes client configuration {"v": 0}
2024-06-11 12:33:21.980 W0611 10:33:21.980813       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-06-11 12:33:21.981 2024-06-11T10:33:21Z    info    setup   Starting standalone health check server {"v": 0}
2024-06-11 12:33:21.981 2024-06-11T10:33:21Z    info    setup   Getting the kong admin api client configuration {"v": 0}
2024-06-11 12:33:21.981 2024-06-11T10:33:21Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "0/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-06-11 12:33:22.983 2024-06-11T10:33:22Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "1/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-06-11 12:33:23.985 2024-06-11T10:33:23Z    info    setup   Retrying kong admin api client call after error {"v": 0, "retries": "2/60", "error": "making HTTP request: Get \"http://localhost:8001/\": dial tcp [::1]:8001: connect: connection refused"}
2024-06-11 12:33:30.032 2024-06-11T10:33:30Z    info    setup   Configuring and building the controller manager {"v": 0}
2024-06-11 12:33:30.032 2024-06-11T10:33:30Z    info    setup   Building the manager runtime scheme and loading apis into the scheme    {"v": 0}
2024-06-11 12:33:30.035 2024-06-11T10:33:30Z    info    setup   Database mode detected, enabling leader election    {"v": 0}
2024-06-11 12:33:30.035 2024-06-11T10:33:30Z    info    setup   Manager set up with multiple namespaces {"v": 0, "namespaces": ["my-company"]}
2024-06-11 12:33:30.046 2024-06-11T10:33:30Z    info    setup   Initializing Dataplane Client   {"v": 0}
2024-06-11 12:33:30.046 2024-06-11T10:33:30Z    info    setup   Emitting Kubernetes events enabled, creating an event recorder for kong-client  {"v": 0}
2024-06-11 12:33:30.046 2024-06-11T10:33:30Z    info    setup   Starting Admission Server   {"v": 0}
2024-06-11 12:33:30.047 2024-06-11T10:33:30Z    info    controller-runtime.certwatcher  Updated current TLS certificate {"v": 0}
2024-06-11 12:33:30.047 2024-06-11T10:33:30Z    info    setup   Initializing Dataplane Synchronizer {"v": 0}
2024-06-11 12:33:30.047 2024-06-11T10:33:30Z    info    controller-runtime.certwatcher  Starting certificate watcher    {"v": 0}
2024-06-11 12:33:30.047 2024-06-11T10:33:30Z    info    setup   Starting Status Updater {"v": 0}
2024-06-11 12:33:30.047 2024-06-11T10:33:30Z    info    setup   Initializing Dataplane address Discovery    {"v": 0}
2024-06-11 12:33:30.047 2024-06-11T10:33:30Z    info    setup   Falling back to a default address finder for UDP    {"v": 0, "reason": "no publish status address or publish service were provided"}
2024-06-11 12:33:30.047 2024-06-11T10:33:30Z    info    setup   Starting Enabled Controllers    {"v": 0}
2024-06-11 12:33:30.062 2024-06-11T10:33:30Z    info    controllers.Dynamic/Gateway Required CustomResourceDefinitions are not installed, setting up a watch for them in case they are installed afterward  {"v": 0}
2024-06-11 12:33:30.063 2024-06-11T10:33:30Z    info    controllers.Dynamic/HTTPRoute   Required CustomResourceDefinitions are not installed, setting up a watch for them in case they are installed afterward  {"v": 0}
2024-06-11 12:33:30.065 2024-06-11T10:33:30Z    info    controllers.Dynamic/ReferenceGrant  Required CustomResourceDefinitions are not installed, setting up a watch for them in case they are installed afterward  {"v": 0}
2024-06-11 12:33:30.065 2024-06-11T10:33:30Z    info    setup   Add readiness probe to health server    {"v": 0}
2024-06-11 12:33:30.065 2024-06-11T10:33:30Z    info    setup   Starting KongLicense controller {"v": 0}
2024-06-11 12:33:30.067 2024-06-11T10:33:30Z    info    setup   Inject license getter to config translator  {"v": 0, "license_getter_type": "*configuration.KongV1Alpha1KongLicenseReconciler"}
2024-06-11 12:33:30.086 2024-06-11T10:33:30Z    info    telemetry   starting telemetry manager  {"v": 0}
2024-06-11 12:33:30.086 2024-06-11T10:33:30Z    info    setup   Anonymous reports enabled   {"v": 0}
2024-06-11 12:33:30.086 2024-06-11T10:33:30Z    info    setup   Starting manager    {"v": 0}
2024-06-11 12:33:30.086 2024-06-11T10:33:30Z    info    controller-runtime.metrics  Starting metrics server {"v": 0}
2024-06-11 12:33:30.086 2024-06-11T10:33:30Z    info    controller-runtime.metrics  Serving metrics server  {"v": 0, "bindAddress": ":10255", "secure": false}
2024-06-11 12:33:30.187 I0611 10:33:30.187200       1 leaderelection.go:250] attempting to acquire leader lease my-company/kong-ingress-controller-leader-my-company-kong...
2024-06-11 12:33:46.126 I0611 10:33:46.126479       1 leaderelection.go:260] successfully acquired lease my-company/kong-ingress-controller-leader-my-company-kong
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Ingress.netv1   Starting EventSource    {"v": 0, "source": "channel source: 0x40009e4ac0"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.EndpointSlice   Starting EventSource    {"v": 0, "source": "kind source: *v1.EndpointSlice"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.EndpointSlice   Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Ingress.netv1   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Ingress.netv1   Starting EventSource    {"v": 0, "source": "kind source: *v1.Ingress"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Ingress.netv1   Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Secrets Starting EventSource    {"v": 0, "source": "kind source: *v1.Secret"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongIngress Starting EventSource    {"v": 0, "source": "kind source: *v1.KongIngress"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "channel source: 0x40009e4cc0"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongIngress Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.TCPIngress"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.TCPIngress  Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Secrets Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.IngressClass.netv1  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.IngressClass.netv1  Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Service Starting EventSource    {"v": 0, "source": "kind source: *v1.Service"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Service Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongPlugin  Starting EventSource    {"v": 0, "source": "kind source: *v1.KongPlugin"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongPlugin  Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongConsumer    Starting EventSource    {"v": 0, "source": "channel source: 0x40009e4e00"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongConsumer    Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "channel source: 0x40009e4c40"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongConsumer    Starting EventSource    {"v": 0, "source": "kind source: *v1.KongConsumer"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongConsumer    Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.UDPIngress"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongClusterPlugin   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongClusterPlugin   Starting EventSource    {"v": 0, "source": "kind source: *v1.KongClusterPlugin"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongClusterPlugin   Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.UDPIngress  Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.IngressClassParameters  Starting EventSource    {"v": 0, "source": "kind source: *v1alpha1.IngressClassParameters"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.IngressClassParameters  Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1.Service"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Dynamic/Gateway Starting EventSource    {"v": 0, "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongUpstreamPolicy"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.Dynamic/Gateway Starting Controller {"v": 0}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "channel source: 0x4000c56200"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "channel source: 0x4000c56240"}
2024-06-11 12:33:46.127 2024-06-11T10:33:46Z    info    controllers.KongUpstreamPolicy  Starting Controller {"v": 0}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongVault   Starting EventSource    {"v": 0, "source": "channel source: 0x4000c56280"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongVault   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongVault   Starting EventSource    {"v": 0, "source": "kind source: *v1alpha1.KongVault"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongVault   Starting Controller {"v": 0}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.Dynamic/ReferenceGrant  Starting EventSource    {"v": 0, "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.Dynamic/ReferenceGrant  Starting Controller {"v": 0}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.Dynamic/HTTPRoute   Starting EventSource    {"v": 0, "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.Dynamic/HTTPRoute   Starting Controller {"v": 0}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongLicense Starting EventSource    {"v": 0, "source": "channel source: 0x4000c56800"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongLicense Starting EventSource    {"v": 0, "source": "kind source: *v1alpha1.KongLicense"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongLicense Starting Controller {"v": 0}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "channel source: 0x40009e4e80"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongConsumerGroup"}
2024-06-11 12:33:46.128 2024-06-11T10:33:46Z    info    controllers.KongConsumerGroup   Starting Controller {"v": 0}
2024-06-11 12:33:46.152 W0611 10:33:46.151085       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:46.152 E0611 10:33:46.151127       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:47.381 W0611 10:33:47.381020       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:47.381 E0611 10:33:47.381055       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:49.529 W0611 10:33:49.528732       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:49.529 E0611 10:33:49.529020       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:52.866 W0611 10:33:52.866205       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:52.866 E0611 10:33:52.866502       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:33:54.267 2024-06-11T10:33:54Z    info    Successfully synced configuration to Kong   {"url": "http://localhost:8001", "update_strategy": "DBMode", "v": 0}
2024-06-11 12:34:00.092 time="2024-06-11T10:34:00Z" level=error msg="failed to forward report" error="failed to connect to reporting server: dial tcp 44.221.159.161:61833: i/o timeout" forwarder=TLSForwarder
2024-06-11 12:34:02.215 W0611 10:34:02.215528       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:34:02.215 E0611 10:34:02.215585       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:34:25.289 W0611 10:34:25.288829       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:34:25.289 E0611 10:34:25.289209       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:35:07.165 W0611 10:35:07.165101       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:35:07.165 E0611 10:35:07.165594       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.EndpointSlice   Could not wait for Cache to sync    {"error": "failed to wait for DiscoveryV1EndpointSlice caches to sync: timed out waiting for cache to be synced for Kind *v1.EndpointSlice"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.Ingress.netv1   Could not wait for Cache to sync    {"error": "failed to wait for NetV1Ingress caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.KongIngress Could not wait for Cache to sync    {"error": "failed to wait for KongV1KongIngress caches to sync: timed out waiting for cache to be synced for Kind *v1.KongIngress"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.Secrets Could not wait for Cache to sync    {"error": "failed to wait for CoreV1Secret caches to sync: timed out waiting for cache to be synced for Kind *v1.Secret"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.IngressClass.netv1  Could not wait for Cache to sync    {"error": "failed to wait for NetV1IngressClass caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.TCPIngress  Could not wait for Cache to sync    {"error": "failed to wait for KongV1Beta1TCPIngress caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.KongPlugin  Could not wait for Cache to sync    {"error": "failed to wait for KongV1KongPlugin caches to sync: timed out waiting for cache to be synced for Kind *v1.KongPlugin"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.Service Could not wait for Cache to sync    {"error": "failed to wait for CoreV1Service caches to sync: timed out waiting for cache to be synced for Kind *v1.Service"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.KongClusterPlugin   Could not wait for Cache to sync    {"error": "failed to wait for KongV1KongClusterPlugin caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.KongConsumer    Could not wait for Cache to sync    {"error": "failed to wait for KongV1KongConsumer caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.IngressClassParameters  Could not wait for Cache to sync    {"error": "failed to wait for KongV1Alpha1IngressClassParameters caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.IngressClassParameters"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.Dynamic/Gateway Could not wait for Cache to sync    {"error": "failed to wait for DynamicCRDController caches to sync: timed out waiting for cache to be synced for Kind *v1.CustomResourceDefinition"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   controllers.UDPIngress  Could not wait for Cache to sync    {"error": "failed to wait for KongV1Beta1UDPIngress caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    info    Stopping and waiting for non leader election runnables  {"v": 0}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    info    Stopping and waiting for leader election runnables  {"v": 0}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for NetV1Ingress caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    info    controllers.Dynamic/HTTPRoute   Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1KongIngress caches to sync: timed out waiting for cache to be synced for Kind *v1.KongIngress"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for CoreV1Secret caches to sync: timed out waiting for cache to be synced for Kind *v1.Secret"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    info    controllers.KongVault   Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    info    dataplane-synchronizer  Context done: shutting down the proxy update server {"v": 0}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for NetV1IngressClass caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1Beta1TCPIngress caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.128 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1KongPlugin caches to sync: timed out waiting for cache to be synced for Kind *v1.KongPlugin"}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    info    controllers.Dynamic/ReferenceGrant  Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    error   controller-runtime.source.EventHandler  failed to get informer from cache   {"error": "Timeout: failed waiting for *v1alpha1.KongLicense Informer to sync"}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    info    controllers.KongLicense Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    error   controllers.KongUpstreamPolicy  Could not wait for Cache to sync    {"error": "failed to wait for KongUpstreamPolicy caches to sync: timed out waiting for cache to be synced for Kind *v1.Service"}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for CoreV1Service caches to sync: timed out waiting for cache to be synced for Kind *v1.Service"}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1KongClusterPlugin caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1KongConsumer caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.129 2024-06-11T10:35:46Z    info    controllers.Dynamic/ReferenceGrant  Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.KongConsumerGroup   Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.KongConsumerGroup   Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1Alpha1IngressClassParameters caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.IngressClassParameters"}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for DynamicCRDController caches to sync: timed out waiting for cache to be synced for Kind *v1.CustomResourceDefinition"}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1Beta1UDPIngress caches to sync: timed out waiting for cache to be synced for Kind *v1.IngressClass"}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongUpstreamPolicy caches to sync: timed out waiting for cache to be synced for Kind *v1.Service"}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.Dynamic/HTTPRoute   Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.KongLicense Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.Dynamic/HTTPRoute   All workers finished    {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.KongVault   Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.KongVault   All workers finished    {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.Dynamic/ReferenceGrant  All workers finished    {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.KongLicense All workers finished    {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    controllers.KongConsumerGroup   All workers finished    {"v": 0}
2024-06-11 12:35:46.130 2024-06-11T10:35:46Z    info    Stopping and waiting for caches {"v": 0}
2024-06-11 12:35:46.131 2024-06-11T10:35:46Z    info    Stopping and waiting for webhooks   {"v": 0}
2024-06-11 12:35:46.131 2024-06-11T10:35:46Z    info    Stopping and waiting for HTTP servers   {"v": 0}
2024-06-11 12:35:46.131 W0611 10:35:46.128582       1 reflector.go:462] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-11 12:35:46.131 2024-06-11T10:35:46Z    info    controller-runtime.metrics  Shutting down metrics server with timeout of 1 minute   {"v": 0}
2024-06-11 12:35:46.131 2024-06-11T10:35:46Z    info    Wait completed, proceeding to shutdown the manager  {"v": 0}
2024-06-11 12:35:46.131 2024-06-11T10:35:46Z    info    telemetry   stopping telemetry manager  {"v": 0}
2024-06-11 12:35:46.131 Error: failed to wait for DiscoveryV1EndpointSlice caches to sync: timed out waiting for cache to be synced for Kind *v1.EndpointSlice
2024-06-11 12:35:46.737 2024-06-11T10:35:46Z    info    Diagnostics server disabled {"v": 0}
2024-06-11 12:35:46.737 2024-06-11T10:35:46Z    info    setup   Starting controller manager {"v": 0, "release": "3.1.5", "repo": "https://github.com/Kong/kubernetes-ingress-controller.git", "commit": "d3e401dcb86ecc40a828eaf5da20e6c90a5bf323"}
2024-06-11 12:35:46.737 2024-06-11T10:35:46Z    info    setup   The ingress class name has been set {"v": 0, "value": "my-company-kong"}
2024-06-11 12:35:46.737 2024-06-11T10:35:46Z    info    setup   Getting enabled options and features    {"v": 0}
2024-06-11 12:35:46.737 2024-06-11T10:35:46Z    info    setup   Getting the kubernetes client configuration {"v": 0}
2024-06-11 12:35:46.737 W0611 10:35:46.737390       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-06-11 12:35:46.737 2024-06-11T10:35:46Z    info    setup   Starting standalone health check server {"v": 0}
2024-06-11 12:35:46.737 2024-06-11T10:35:46Z    info    setup   Getting the kong admin api client configuration {"v": 0}
2024-06-11 12:35:46.764 2024-06-11T10:35:46Z    info    setup   Configuring and building the controller manager {"v": 0}
2024-06-11 12:35:46.764 2024-06-11T10:35:46Z    info    setup   Building the manager runtime scheme and loading apis into the scheme    {"v": 0}
2024-06-11 12:35:46.767 2024-06-11T10:35:46Z    info    setup   Database mode detected, enabling leader election    {"v": 0}
2024-06-11 12:35:46.767 2024-06-11T10:35:46Z    info    setup   Manager set up with multiple namespaces {"v": 0, "namespaces": ["my-company"]}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    setup   Initializing Dataplane Client   {"v": 0}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    setup   Emitting Kubernetes events enabled, creating an event recorder for kong-client  {"v": 0}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    setup   Starting Admission Server   {"v": 0}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    controller-runtime.certwatcher  Updated current TLS certificate {"v": 0}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    setup   Initializing Dataplane Synchronizer {"v": 0}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    setup   Starting Status Updater {"v": 0}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    setup   Initializing Dataplane address Discovery    {"v": 0}
2024-06-11 12:35:46.780 2024-06-11T10:35:46Z    info    setup   Falling back to a default address finder for UDP    {"v": 0, "reason": "no publish status address or publish service were provided"}
2024-06-11 12:35:46.781 2024-06-11T10:35:46Z    info    setup   Starting Enabled Controllers    {"v": 0}
2024-06-11 12:35:46.781 2024-06-11T10:35:46Z    info    controller-runtime.certwatcher  Starting certificate watcher    {"v": 0}
2024-06-11 12:35:46.798 2024-06-11T10:35:46Z    info    controllers.Dynamic/Gateway Required CustomResourceDefinitions are not installed, setting up a watch for them in case they are installed afterward  {"v": 0}
2024-06-11 12:35:46.800 2024-06-11T10:35:46Z    info    controllers.Dynamic/HTTPRoute   Required CustomResourceDefinitions are not installed, setting up a watch for them in case they are installed afterward  {"v": 0}
2024-06-11 12:35:46.801 2024-06-11T10:35:46Z    info    controllers.Dynamic/ReferenceGrant  Required CustomResourceDefinitions are not installed, setting up a watch for them in case they are installed afterward  {"v": 0}
2024-06-11 12:35:46.801 2024-06-11T10:35:46Z    info    setup   Add readiness probe to health server    {"v": 0}
2024-06-11 12:35:46.801 2024-06-11T10:35:46Z    info    setup   Starting KongLicense controller {"v": 0}
2024-06-11 12:35:46.803 2024-06-11T10:35:46Z    info    setup   Inject license getter to config translator  {"v": 0, "license_getter_type": "*configuration.KongV1Alpha1KongLicenseReconciler"}
2024-06-11 12:35:46.824 2024-06-11T10:35:46Z    info    telemetry   starting telemetry manager  {"v": 0}
2024-06-11 12:35:46.825 2024-06-11T10:35:46Z    info    setup   Anonymous reports enabled   {"v": 0}
2024-06-11 12:35:46.825 2024-06-11T10:35:46Z    info    setup   Starting manager    {"v": 0}
2024-06-11 12:35:46.825 2024-06-11T10:35:46Z    info    controller-runtime.metrics  Starting metrics server {"v": 0}
2024-06-11 12:35:46.825 2024-06-11T10:35:46Z    info    controller-runtime.metrics  Serving metrics server  {"v": 0, "bindAddress": ":10255", "secure": false}
2024-06-11 12:35:46.926 I0611 10:35:46.926055       1 leaderelection.go:250] attempting to acquire leader lease my-company/kong-ingress-controller-leader-my-company-kong...
2024-06-11 12:36:03.434 I0611 10:36:03.433326       1 leaderelection.go:260] successfully acquired lease my-company/kong-ingress-controller-leader-my-company-kong
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Ingress.netv1   Starting EventSource    {"v": 0, "source": "channel source: 0x40006b2200"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.IngressClass.netv1  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Ingress.netv1   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Ingress.netv1   Starting EventSource    {"v": 0, "source": "kind source: *v1.Ingress"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Ingress.netv1   Starting Controller {"v": 0}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "channel source: 0x40006b2380"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Secrets Starting EventSource    {"v": 0, "source": "kind source: *v1.Secret"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.UDPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.UDPIngress"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Secrets Starting Controller {"v": 0}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.EndpointSlice   Starting EventSource    {"v": 0, "source": "kind source: *v1.EndpointSlice"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.EndpointSlice   Starting Controller {"v": 0}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Service Starting EventSource    {"v": 0, "source": "kind source: *v1.Service"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.IngressClass.netv1  Starting Controller {"v": 0}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.Service Starting Controller {"v": 0}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.UDPIngress  Starting Controller {"v": 0}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.KongIngress Starting EventSource    {"v": 0, "source": "kind source: *v1.KongIngress"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.KongIngress Starting Controller {"v": 0}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.KongPlugin  Starting EventSource    {"v": 0, "source": "kind source: *v1.KongPlugin"}
2024-06-11 12:36:03.434 2024-06-11T10:36:03Z    info    controllers.KongPlugin  Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1.Service"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.Dynamic/HTTPRoute   Starting EventSource    {"v": 0, "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongUpstreamPolicy"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.Dynamic/HTTPRoute   Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "channel source: 0x4000954f40"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongUpstreamPolicy  Starting EventSource    {"v": 0, "source": "channel source: 0x4000954f80"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongUpstreamPolicy  Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongVault   Starting EventSource    {"v": 0, "source": "channel source: 0x4000954fc0"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongVault   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongVault   Starting EventSource    {"v": 0, "source": "kind source: *v1alpha1.KongVault"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "channel source: 0x40006b2400"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.TCPIngress  Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.TCPIngress"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongVault   Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.TCPIngress  Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.IngressClassParameters  Starting EventSource    {"v": 0, "source": "kind source: *v1alpha1.IngressClassParameters"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.IngressClassParameters  Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.Dynamic/Gateway Starting EventSource    {"v": 0, "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.Dynamic/Gateway Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.Dynamic/ReferenceGrant  Starting EventSource    {"v": 0, "source": "kind source: *v1.CustomResourceDefinition"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.Dynamic/ReferenceGrant  Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumer    Starting EventSource    {"v": 0, "source": "channel source: 0x40006b2540"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongClusterPlugin   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongClusterPlugin   Starting EventSource    {"v": 0, "source": "kind source: *v1.KongClusterPlugin"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongClusterPlugin   Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumer    Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumer    Starting EventSource    {"v": 0, "source": "kind source: *v1.KongConsumer"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "channel source: 0x40006b25c0"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumer    Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "kind source: *v1.IngressClass"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumerGroup   Starting EventSource    {"v": 0, "source": "kind source: *v1beta1.KongConsumerGroup"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongLicense Starting EventSource    {"v": 0, "source": "channel source: 0x40009aeb00"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongLicense Starting EventSource    {"v": 0, "source": "kind source: *v1alpha1.KongLicense"}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongLicense Starting Controller {"v": 0}
2024-06-11 12:36:03.435 2024-06-11T10:36:03Z    info    controllers.KongConsumerGroup   Starting Controller {"v": 0}
2024-06-11 12:36:03.471 W0611 10:36:03.470971       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:03.471 E0611 10:36:03.471024       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:04.328 W0611 10:36:04.328382       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:04.328 E0611 10:36:04.328462       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:06.975 W0611 10:36:06.975197       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:06.975 E0611 10:36:06.975497       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:11.584 2024-06-11T10:36:11Z    info    Successfully synced configuration to Kong   {"url": "http://localhost:8001", "update_strategy": "DBMode", "v": 0}
2024-06-11 12:36:12.291 W0611 10:36:12.291470       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:12.291 E0611 10:36:12.291522       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:16.832 time="2024-06-11T10:36:16Z" level=error msg="failed to forward report" error="failed to connect to reporting server: dial tcp 34.203.59.233:61833: i/o timeout" forwarder=TLSForwarder
2024-06-11 12:36:20.922 W0611 10:36:20.922517       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:20.922 E0611 10:36:20.922573       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:44.030 W0611 10:36:44.029942       1 reflector.go:539] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:36:44.030 E0611 10:36:44.030382       1 reflector.go:147] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: Failed to watch *v1alpha1.KongLicense: failed to list *v1alpha1.KongLicense: konglicenses.configuration.konghq.com is forbidden: User "system:serviceaccount:my-company:my-company-kong-kong" cannot list resource "konglicenses" in API group "configuration.konghq.com" at the cluster scope
2024-06-11 12:37:04.550 2024-06-11T10:37:04Z    info    Signal received, shutting down  {"v": 0, "graceful_period": "0s", "signal": "terminated"}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    setup.health-check  Healthz server closed   {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    Stopping and waiting for non leader election runnables  {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    Stopping and waiting for leader election runnables  {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    dataplane-synchronizer  Context done: shutting down the proxy update server {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.TCPIngress  Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.TCPIngress  Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.KongConsumer    Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.TCPIngress  All workers finished    {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.KongConsumer    Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.Dynamic/HTTPRoute   Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.Dynamic/HTTPRoute   Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.551 2024-06-11T10:37:04Z    info    controllers.Dynamic/HTTPRoute   All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Ingress.netv1   Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.EndpointSlice   Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Ingress.netv1   Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.EndpointSlice   Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Secrets Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Secrets Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Secrets All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.IngressClassParameters  Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.IngressClassParameters  Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.IngressClassParameters  All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Dynamic/Gateway Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Dynamic/Gateway Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Dynamic/Gateway All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Dynamic/ReferenceGrant  Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Dynamic/ReferenceGrant  Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Dynamic/ReferenceGrant  All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    error   controller-runtime.source.EventHandler  failed to get informer from cache   {"error": "Timeout: failed waiting for *v1alpha1.KongLicense Informer to sync"}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    error   controllers.KongVault   Could not wait for Cache to sync    {"error": "failed to wait for KongV1Alpha1KongVault caches to sync: cache did not sync"}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1Alpha1KongVault caches to sync: cache did not sync"}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.IngressClass.netv1  Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.IngressClass.netv1  Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    error   controllers.UDPIngress  Could not wait for Cache to sync    {"error": "failed to wait for KongV1Beta1UDPIngress caches to sync: cache did not sync"}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1Beta1UDPIngress caches to sync: cache did not sync"}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Service Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.Service Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.IngressClass.netv1  All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongIngress Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongIngress Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongIngress All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongPlugin  Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongPlugin  Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    error   controllers.KongClusterPlugin   Could not wait for Cache to sync    {"error": "failed to wait for KongV1KongClusterPlugin caches to sync: cache did not sync"}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    error   error received after stop sequence was engaged  {"error": "failed to wait for KongV1KongClusterPlugin caches to sync: cache did not sync"}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongUpstreamPolicy  Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongUpstreamPolicy  Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongUpstreamPolicy  All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongConsumerGroup   Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongConsumerGroup   Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongConsumerGroup   All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongConsumer    All workers finished    {"v": 0}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongLicense Starting workers    {"v": 0, "worker count": 1}
2024-06-11 12:37:04.553 2024-06-11T10:37:04Z    info    controllers.KongLicense Shutdown signal received, waiting for all workers to finish {"v": 0}
2024-06-11 12:37:04.554 2024-06-11T10:37:04Z    info    controllers.KongLicense All workers finished    {"v": 0}
2024-06-11 12:37:04.555 2024-06-11T10:37:04Z    info    controllers.EndpointSlice   All workers finished    {"v": 0}
2024-06-11 12:37:04.555 2024-06-11T10:37:04Z    info    controllers.KongPlugin  All workers finished    {"v": 0}
2024-06-11 12:37:04.585 2024-06-11T10:37:04Z    info    controllers.Ingress.netv1   All workers finished    {"v": 0}
2024-06-11 12:37:04.585 2024-06-11T10:37:04Z    info    controllers.Service All workers finished    {"v": 0}
2024-06-11 12:37:04.585 2024-06-11T10:37:04Z    info    Stopping and waiting for caches {"v": 0}
2024-06-11 12:37:04.586 W0611 10:37:04.585641       1 reflector.go:462] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: watch of *v1.Ingress ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-11 12:37:04.586 2024-06-11T10:37:04Z    info    Stopping and waiting for webhooks   {"v": 0}
2024-06-11 12:37:04.586 2024-06-11T10:37:04Z    info    Stopping and waiting for HTTP servers   {"v": 0}
2024-06-11 12:37:04.586 W0611 10:37:04.585721       1 reflector.go:462] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-11 12:37:04.586 2024-06-11T10:37:04Z    info    controller-runtime.metrics  Shutting down metrics server with timeout of 1 minute   {"v": 0}
2024-06-11 12:37:04.586 W0611 10:37:04.585908       1 reflector.go:462] pkg/mod/k8s.io/client-go@v0.29.1/tools/cache/reflector.go:229: watch of *v1.KongClusterPlugin ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
2024-06-11 12:37:04.586 2024-06-11T10:37:04Z    info    Wait completed, proceeding to shutdown the manager  {"v": 0}
2024-06-11 12:37:04.586 2024-06-11T10:37:04Z    error   error received after stop sequence was engaged  {"error": "leader election lost"}
2024-06-11 12:37:04.586 2024-06-11T10:37:04Z    info    telemetry   stopping telemetry manager  {"v": 0}
pmalek commented 3 months ago

Hi @sebboer,

What KIC version do you use? Can you post your values.yaml?

The change that adds the RBAC policy rules for this should already be included in kong 2.37.1 https://github.com/Kong/charts/commit/cb4483006a336d92fc5a3ddee7532e12bf9bc612 https://github.com/Kong/charts/blob/kong-2.37.1/charts/kong/templates/_helpers.tpl#L1644-L1661

sebboer commented 3 months ago

Coming from 2.36.0 I'm using KIC 3.0 that gets updated to 3.1.0 Changelog

values.yaml

admin:
  enabled: true
  http:
    enabled: true
  tls:
    enabled: false
  type: ClusterIP
affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchLabels:
            app.kubernetes.io/instance: kong
            app.kubernetes.io/name: kong
        topologyKey: kubernetes.io/hostname
env:
  NGINX_HTTP_CLIENT_MAX_BODY_SIZE: 0
  NGINX_HTTP_PROXY_HTTP_VERSION: "1.1"
  NGINX_PROXY_CLIENT_MAX_BODY_SIZE: 0
  NGINX_PROXY_PROXY_HTTP_VERSION: "1.1"
  PROXY_LISTEN: 0.0.0.0:8000, 0.0.0.0:8443 ssl
  TRUSTED_IPS: 10.0.0.0/8
  database: postgres
ingressController:
  enabled: true
  ingressClass: my-company-kong
  installCRDs: false
  watchNamespaces:
    - my-company
manager:
  enabled: false
nodeSelector:
  kubernetes.io/arch: arm64
  node.kubernetes.io/role: agent
portal:
  enabled: false
serviceMonitor:
  enabled: true
  namespace: monitoring
pmalek commented 3 months ago

We didn't account for watchNamespaces when we updated the policy RBAC rules for KongLicenses.

1084 fixes that.

Before next version of the chart gets released you can disable the license controller (if you're not using its functionality .

sebboer commented 3 months ago

Thank you! The update now works after adding --enable-controller-kong-license=false.

pmalek commented 3 months ago

I'm glad that this is solved 👍

In case you'd like to test the proper fix, it will be released in the next version of kong helm chart.