Kong / charts

Helm chart for Kong
Apache License 2.0
239 stars 473 forks source link

`helm template kong kong/ingress` does not produce the same output like `helm template kong kong/ingress --validate` #1049

Closed diefans closed 3 months ago

diefans commented 5 months ago

Is there an existing issue for this?

Current Behavior

When you call helm template with --validate the generated output contains extra rules for the ClusterRole, which are needed to run the ingress controller:

- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - gateways
  verbs:
  - get
  - list
  - update
  - watch
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - gateways/status
  verbs:
  - get
  - update
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - httproutes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - httproutes/status
  verbs:
  - get
  - update
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - referencegrants
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - referencegrants/status
  verbs:
  - get
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - tcproutes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - tcproutes/status
  verbs:
  - get
  - update
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - tlsroutes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - tlsroutes/status
  verbs:
  - get
  - update
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - udproutes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - udproutes/status
  verbs:
  - get
  - update
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - grpcroutes
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - gateway.networking.k8s.io
  resources:
  - grpcroutes/status
  verbs:
  - get
  - patch
  - update

Expected Behavior

The extra rules generated by --validate should be generated without calling --validate.

Steps To Reproduce

(
   with_validate=$(mktemp);
   without_validate=$(mktemp);
   helm template kong kong/ingress -n kong --validate > ${with_validate};
   helm template kong kong/ingress -n kong > ${without_validate};
   diff -u ${without_validate} ${with_validate};
   rm ${with_validate} ${without_validate};
)

Kong Ingress Controller version

apiVersion: v2
appVersion: "3.4"
dependencies:
- alias: controller
  condition: controller.enabled
  name: kong
  repository: https://charts.konghq.com
  version: '>=2.33.3'
- alias: gateway
  condition: gateway.enabled
  name: kong
  repository: https://charts.konghq.com
  version: '>=2.33.3'
description: Deploy Kong Ingress Controller and Kong Gateway
home: https://konghq.com/
icon: https://s3.amazonaws.com/downloads.kong/universe/assets/icon-kong-inc-large.png
maintainers:
- email: team-k8s@konghq.com
  name: team-k8s-bot
name: ingress
sources:
- https://github.com/Kong/charts/tree/main/charts/ingress
version: 0.10.2

Kubernetes version

Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0

helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"", GoVersion:"go1.17.13"}

Anything else?

The actual problem when using --validate is, that it inspects the state of the cluster, and for people using other tools to install kong/ingress (as also suggested in https://github.com/Kong/kubernetes-ingress-controller/issues/4712) the generation of the manifests depends on the cluster state, e.g. if you have a dev-cluster and use tools like tilt - it effectively prevents calling helm template --validate, since helm is throwing an error like:

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "kong-controller" in namespace "kong" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: key "app.kubernetes.io/managed-by" must equal "Helm": current value is "tilt"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "kong"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "kong"
pmalek commented 3 months ago

Hi @diefans,

The reason for the above is that helm template ... without --validate doesn't consult the api server for served APIs which makes our template logic in e.g. https://github.com/Kong/charts/blob/e09fd7a54ba99466328ccf9a581e50a3d516f58c/charts/kong/templates/_helpers.tpl#L1502 return false.

If you want to get similar results between the 2 invocations you can provide the Gateway API as such:

helm template kong --api-versions gateway.networking.k8s.io/v1 ...

and you should get a similar result to the one that you get with --validate (assuming that this is the API that's served by your cluster.

I'm going to close this issue now but feel free to reach out again if there's still something to cover.