search

Kong / charts

Helm chart for Kong
Apache License 2.0
242 stars 474 forks source link

Ingress controller fails to start up after updating from 2.30.0 to 2.39.3 #1102

Closed A7exSchin closed 1 month ago

A7exSchin commented 1 month ago

Hi there!

We recently updated our kong/ingress chart from 0.10.1 to 0.13.1. This in turn updated the kong/kong chart form 2.30.0 to 2.39.3.

I checked the values and it did not look like anything affecting our deployment was changed. However, the ingress controller restarts every 3 minutes with multiple errors, which are not really indicative of what is wrong.

First error message block:

2024-07-09T09:12:53Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:13:03Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:13:13Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:13:23Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:13:33Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:13:43Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:13:53Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024/07/09 09:14:03 http: TLS handshake error from 10.1.2.125:54310: EOF
2024-07-09T09:14:03Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024/07/09 09:14:05 http: TLS handshake error from 10.1.0.19:33548: EOF
2024-07-09T09:14:13Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:14:23Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:14:33Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:14:43Z   error   controller-runtime.source.EventHandler  if kind is a CRD, it should be installed before calling Start  {"kind": "KongVault.configuration.konghq.com", "error": "no matches for kind \"KongVault\" in version \"configuration.konghq.com/v1alpha1\""}
2024-07-09T09:14:44Z   error   controllers.KongVault  Could not wait for Cache to sync {"error": "failed to wait for KongV1Alpha1KongVault caches to sync: timed out waiting for cache to be synced for Kind *v1alpha1.KongVault"}

We are not using KongVaults at all in our deployment.

This is the second error log:

W0709 09:14:44.362408       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.CustomResourceDefinition ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362472       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.Secret ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362512       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.KongPlugin ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362547       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1beta1.UDPIngress ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362585       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.KongIngress ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362619       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.Ingress ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362656       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.KongConsumer ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362740       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.Gateway ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362781       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1beta1.KongUpstreamPolicy ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362832       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.GatewayClass ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362861       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1beta1.ReferenceGrant ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362888       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.KongClusterPlugin ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362944       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1alpha1.IngressClassParameters ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362971       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1beta1.KongConsumerGroup ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.362997       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1beta1.TCPIngress ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.363040       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.IngressClass ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.363094       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.EndpointSlice ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.363124       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.Service ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
W0709 09:14:44.363151       1 reflector.go:470] pkg/mod/k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232: watch of *v1.HTTPRoute ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding

This is the code used to deploy the controller inside the helm/ingress chart:

controller:
  enabled: true

  deployment:
    kong:
      enabled: false

  proxy:
    nameOverride: "{{ .Release.Name }}-gateway-proxy"

  ingressController:
    enabled: true

    gatewayDiscovery:
      enabled: true
      generateAdminApiService: true

  podAnnotations:
    kuma.io/gateway: enabled
    # This port must match your Kong admin API port. 8444 is the default.
    # If you set gateway.admin.tls.containerPort, change these annotations
    # to use that value.
    traffic.kuma.io/exclude-outbound-ports: "8444"
    traffic.sidecar.istio.io/excludeOutboundPorts: "8444"
pmalek commented 1 month ago

Hi @A7exSchin 👋

You should get the new CRDs (KongVault specifically) installed when installing kong/ingress 0.13.1.

I just checked it by running

helm upgrade --install --create-namespace -n kong kong kong/ingress --version v0.13.1

and after deleting the CRD before the installation, I do get it installed with the chart.

Not sure why that wouldn't work. Can you share the values.yaml?

You can always disable the vault controller if you don't use it 

controller:
  ingressController:
    env:
      enable_controller_kong_vault: "false"

As for the second error, this is related to an upstream issue https://github.com/kubernetes-sigs/controller-runtime/issues/2723 which is still open.

A7exSchin commented 1 month ago

Hi @pmalek !

The quick response is very much appreciated. The last code block you see is the values.yaml for the controller ressource (as I am using the ingress chart, I only included the nested values.yaml for the nested chart and excluded the part for the gateway).

I will try to disable the kong vault.

We are currently deploying this helm chart via terraform so I cannot really say why it is not creating the CRDs properly. Will try to investigate

A7exSchin commented 1 month ago

Hi again @pmalek !

We disabled the KongVault according to your suggested fix and that worked! Thanks!

However, I am not sure why the Helm Deployment Process is not correctly deploying the CRDs