search

kubernetes-sigs / kustomize

Customization of kubernetes YAML configurations
Apache License 2.0
10.93k stars 2.24k forks source link

HelmChartInflationGenerator does not work for Capabilities.APIVersions in templates #3458

Closed spali closed 3 years ago

spali commented 3 years ago

Describe the bug

Template checking Capabilities.APIVersions fails. It seems that helm via HelmChartInflationGenerator does not get the cluster api's available.

Files that can reproduce the issue

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
  - chartInflator.yaml

chartInflator.yaml

apiVersion: builtin
kind: HelmChartInflationGenerator
metadata:
  name: notImportantHere
chartName: traefik
chartRepoUrl: https://helm.traefik.io/traefik
chartVersion: 9.12.3

releaseName: ingress-traefik
releaseNamespace: ingress-traefik
values: values_ingress.yaml

values_ingress.yaml

ingressClass:
  enabled: true

Expected output

helm chart yaml

Actual output

$ kustomize build .
Error: Error: template: traefik/templates/ingressclass.yaml:7:8: executing "traefik/templates/ingressclass.yaml" at <fail "\n\n ERROR: You must have atleast networking.k8s.io/v1beta1 to use ingressClass">: error calling fail: 

 ERROR: You must have atleast networking.k8s.io/v1beta1 to use ingressClass

Use --debug flag to render out invalid YAML
: failed to run command helm template ingress-traefik /tmp/kustomize-169177316/chart/traefik --namespace ingress-traefik --values values_ingress.yaml

Kustomize version

helm version
version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

Helm version

helm version
version.BuildInfo{Version:"v3.4.1", GitCommit:"c4e74854886b2efe3321e185578e6db9be0a6e29", GitTreeState:"clean", GoVersion:"go1.14.11"}

Platform

Windows 10 WSL2

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.1 LTS"

Additional context

helm directly works:

helm repo add traefik https://helm.traefik.io/traefik
helm repo update
# expected result from kustomize
helm install ingress-traefik traefik/traefik --dry-run --values values_ingress.yaml >res_ingress.yml
# to compare what the template renders differently based on the values
helm install ingress-traefik traefik/traefik --dry-run >res_default.yml

cluster:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.6", GitCommit:"fbf646b339dc52336b55d8ec85c181981b86331a", GitTreeState:"clean", BuildDate:"2020-12-18T12:01:36Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

$ kubectl get nodes
NAME                   STATUS   ROLES    AGE    VERSION
kn01....   Ready    master   2d2h   v1.19.6
kn02....   Ready    <none>   2d2h   v1.19.6
kn03....   Ready    <none>   2d2h   v1.19.6

$ kubectl api-resources -o wide
NAME                              SHORTNAMES   APIGROUP                       NAMESPACED   KIND                             VERBS
bindings                                                                      true         Binding                          [create]
componentstatuses                 cs                                          false        ComponentStatus                  [get list]
configmaps                        cm                                          true         ConfigMap                        [create delete deletecollection get list patch update watch]
endpoints                         ep                                          true         Endpoints                        [create delete deletecollection get list patch update watch]
events                            ev                                          true         Event                            [create delete deletecollection get list patch update watch]
limitranges                       limits                                      true         LimitRange                       [create delete deletecollection get list patch update watch]
namespaces                        ns                                          false        Namespace                        [create delete get list patch update watch]
nodes                             no                                          false        Node                             [create delete deletecollection get list patch update watch]
persistentvolumeclaims            pvc                                         true         PersistentVolumeClaim            [create delete deletecollection get list patch update watch]
persistentvolumes                 pv                                          false        PersistentVolume                 [create delete deletecollection get list patch update watch]
pods                              po                                          true         Pod                              [create delete deletecollection get list patch update watch]
podtemplates                                                                  true         PodTemplate                      [create delete deletecollection get list patch update watch]
replicationcontrollers            rc                                          true         ReplicationController            [create delete deletecollection get list patch update watch]
resourcequotas                    quota                                       true         ResourceQuota                    [create delete deletecollection get list patch update watch]
secrets                                                                       true         Secret                           [create delete deletecollection get list patch update watch]
serviceaccounts                   sa                                          true         ServiceAccount                   [create delete deletecollection get list patch update watch]
services                          svc                                         true         Service                          [create delete get list patch update watch]
mutatingwebhookconfigurations                  admissionregistration.k8s.io   false        MutatingWebhookConfiguration     [create delete deletecollection get list patch update watch]
validatingwebhookconfigurations                admissionregistration.k8s.io   false        ValidatingWebhookConfiguration   [create delete deletecollection get list patch update watch]
customresourcedefinitions         crd,crds     apiextensions.k8s.io           false        CustomResourceDefinition         [create delete deletecollection get list patch update watch]
apiservices                                    apiregistration.k8s.io         false        APIService                       [create delete deletecollection get list patch update watch]
controllerrevisions                            apps                           true         ControllerRevision               [create delete deletecollection get list patch update watch]
daemonsets                        ds           apps                           true         DaemonSet                        [create delete deletecollection get list patch update watch]
deployments                       deploy       apps                           true         Deployment                       [create delete deletecollection get list patch update watch]
replicasets                       rs           apps                           true         ReplicaSet                       [create delete deletecollection get list patch update watch]
statefulsets                      sts          apps                           true         StatefulSet                      [create delete deletecollection get list patch update watch]
tokenreviews                                   authentication.k8s.io          false        TokenReview                      [create]
localsubjectaccessreviews                      authorization.k8s.io           true         LocalSubjectAccessReview         [create]
selfsubjectaccessreviews                       authorization.k8s.io           false        SelfSubjectAccessReview          [create]
selfsubjectrulesreviews                        authorization.k8s.io           false        SelfSubjectRulesReview           [create]
subjectaccessreviews                           authorization.k8s.io           false        SubjectAccessReview              [create]
horizontalpodautoscalers          hpa          autoscaling                    true         HorizontalPodAutoscaler          [create delete deletecollection get list patch update watch]
cronjobs                          cj           batch                          true         CronJob                          [create delete deletecollection get list patch update watch]
jobs                                           batch                          true         Job                              [create delete deletecollection get list patch update watch]
certificatesigningrequests        csr          certificates.k8s.io            false        CertificateSigningRequest        [create delete deletecollection get list patch update watch]
leases                                         coordination.k8s.io            true         Lease                            [create delete deletecollection get list patch update watch]
bgpconfigurations                              crd.projectcalico.org          false        BGPConfiguration                 [delete deletecollection get list patch create update watch]
bgppeers                                       crd.projectcalico.org          false        BGPPeer                          [delete deletecollection get list patch create update watch]
blockaffinities                                crd.projectcalico.org          false        BlockAffinity                    [delete deletecollection get list patch create update watch]
clusterinformations                            crd.projectcalico.org          false        ClusterInformation               [delete deletecollection get list patch create update watch]
felixconfigurations                            crd.projectcalico.org          false        FelixConfiguration               [delete deletecollection get list patch create update watch]
globalnetworkpolicies                          crd.projectcalico.org          false        GlobalNetworkPolicy              [delete deletecollection get list patch create update watch]
globalnetworksets                              crd.projectcalico.org          false        GlobalNetworkSet                 [delete deletecollection get list patch create update watch]
hostendpoints                                  crd.projectcalico.org          false        HostEndpoint                     [delete deletecollection get list patch create update watch]
ipamblocks                                     crd.projectcalico.org          false        IPAMBlock                        [delete deletecollection get list patch create update watch]
ipamconfigs                                    crd.projectcalico.org          false        IPAMConfig                       [delete deletecollection get list patch create update watch]
ipamhandles                                    crd.projectcalico.org          false        IPAMHandle                       [delete deletecollection get list patch create update watch]
ippools                                        crd.projectcalico.org          false        IPPool                           [delete deletecollection get list patch create update watch]
kubecontrollersconfigurations                  crd.projectcalico.org          false        KubeControllersConfiguration     [delete deletecollection get list patch create update watch]
networkpolicies                                crd.projectcalico.org          true         NetworkPolicy                    [delete deletecollection get list patch create update watch]
networksets                                    crd.projectcalico.org          true         NetworkSet                       [delete deletecollection get list patch create update watch]
endpointslices                                 discovery.k8s.io               true         EndpointSlice                    [create delete deletecollection get list patch update watch]
events                            ev           events.k8s.io                  true         Event                            [create delete deletecollection get list patch update watch]
ingresses                         ing          extensions                     true         Ingress                          [create delete deletecollection get list patch update watch]
ingressclasses                                 networking.k8s.io              false        IngressClass                     [create delete deletecollection get list patch update watch]
ingresses                         ing          networking.k8s.io              true         Ingress                          [create delete deletecollection get list patch update watch]
networkpolicies                   netpol       networking.k8s.io              true         NetworkPolicy                    [create delete deletecollection get list patch update watch]
runtimeclasses                                 node.k8s.io                    false        RuntimeClass                     [create delete deletecollection get list patch update watch]
poddisruptionbudgets              pdb          policy                         true         PodDisruptionBudget              [create delete deletecollection get list patch update watch]
podsecuritypolicies               psp          policy                         false        PodSecurityPolicy                [create delete deletecollection get list patch update watch]
clusterrolebindings                            rbac.authorization.k8s.io      false        ClusterRoleBinding               [create delete deletecollection get list patch update watch]
clusterroles                                   rbac.authorization.k8s.io      false        ClusterRole                      [create delete deletecollection get list patch update watch]
rolebindings                                   rbac.authorization.k8s.io      true         RoleBinding                      [create delete deletecollection get list patch update watch]
roles                                          rbac.authorization.k8s.io      true         Role                             [create delete deletecollection get list patch update watch]
priorityclasses                   pc           scheduling.k8s.io              false        PriorityClass                    [create delete deletecollection get list patch update watch]
csidrivers                                     storage.k8s.io                 false        CSIDriver                        [create delete deletecollection get list patch update watch]
csinodes                                       storage.k8s.io                 false        CSINode                          [create delete deletecollection get list patch update watch]
storageclasses                    sc           storage.k8s.io                 false        StorageClass                     [create delete deletecollection get list patch update watch]
volumeattachments                              storage.k8s.io                 false        VolumeAttachment                 [create delete deletecollection get list patch update watch]
middlewares                                    traefik.containo.us            true         Middleware                       [delete deletecollection get list patch create update watch]`

Should be anything to reproduce it, but let me know if you need something else.

spali commented 3 years ago

maybe additional information that could have an impact?: I do not have the KUBECONFIG variable set, nor anything else special. I just have the ~/.kube/config with admin access to the cluster. So helm and kubectl can access without further configuration

spali commented 3 years ago

sorry was a bit unfair ;) I compared to helm install. But the plugin is using helm template. With helm template I need to specify the api also manually with --api-versions networking.k8s.io/v1/IngressClass Which I can also do in the chartInflator.yaml with

extraArgs:
  - "--api-versions=networking.k8s.io/v1/IngressClass"

That the plugin is using helm command template makes sense due it doesn't handle deployment by design. Maybe for the use-case to directly deploy and not only generate it would make sense to offer an option to use install together with --dry-run to not always have to specify the supported api's as extraArgs. But that is a feature request ;) so closing this for documentation.

james-callahan commented 3 years ago

Maybe for the use-case to directly deploy and not only generate it would make sense to offer an option to use install together with --dry-run to not always have to specify the supported api's as extraArgs. But that is a feature request ;) so closing this for documentation.

extraArgs seems to have been removed. How can you pass api-versions now?

brennerm commented 1 year ago

To everyone stumbling across this issue and still looking for a solution. The HelmChartInflationGenerator nowadays supports a dedicated apiVersions parameter that allows you to provide the API versions you want to use.

Example:

namespace: foo

helmCharts:
- name: bar
  releaseName: bar
...
  apiVersions:
    - apiregistration.k8s.io/v1
...