search

pulumi / pulumi-kubernetes

A Pulumi resource provider for Kubernetes to manage API resources and workloads in running clusters
https://www.pulumi.com/docs/reference/clouds/kubernetes/
Apache License 2.0
396 stars 113 forks source link

Helm cache corruption issue #1518

Open headconnect opened 3 years ago

headconnect commented 3 years ago

When attempting to add Azure Key Vault Provider for Secrets Store CSI Driver to my cluster, pulumi failed with an error message stating: "Failed to generate YAML for specified Helm chart: failed to pull chart: no cached repo found. (try 'helm repo update')."

I had not previously pulled the repo with helm, nor had I run helm update.

Expected behavior

The remote chart repo should be added and updated prior to template being executed.

Current behavior

Without adding the chart manuall, it fails with the following error: 

pulumi:pulumi:Stack baseline-k8s-dev.k8s.infratesting create error: Unhandled exception: Error: invocation of kubernetes:helm:template returned an error: failed to generate YAML for specified Helm chart: failed to pull chart: no cached repo found. (try 'helm repo update'):

Steps to reproduce

  1. See guide at https://azure.github.io/secrets-store-csi-driver-provider-azure/getting-started/installation/
  2. Write the following command: (amount of slashes after url has no effect - two used as this is what is present on the pulumi provider examples page) 
    const keyvaultCSI = new k8s.helm.v3.Chart("keyVaultCSI",{
    chart: "csi-secrets-store-provider-azure",
    version: "0.0.17",
    fetchOpts: {
        repo: "https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts//"
    },
    values: {
        logFormatJSON: true,            
    }
},{
    provider: cluster
})
  3. Get error message as above
  4. Run: 
    helm repo add csi-secrets-store-provider-azure https://raw.githubusercontent.com/Azure/secrets-store-csi-driver-provider-azure/master/charts
"csi-secrets-store-provider-azure" has been added to your repositories
  5. Run pulumi again - get same error
  6. Run: 
    λ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "csi-secrets-store-provider-azure" chart repository
...
Update Complete. ⎈Happy Helming!⎈

  7. Run pulumi again - no error:

    
Previewing update (dev.k8s.infratesting):

+  pulumi:pulumi:Stack baseline-k8s-dev.k8s.infratesting create
+  kubernetes:helm.sh/v3:Chart keyVaultCSI create
+  pulumi:pulumi:Stack baseline-k8s-dev.k8s.infratesting create read pulumi:pulumi:StackReference dev.infra.infratesting-reference
+  pulumi:pulumi:Stack baseline-k8s-dev.k8s.infratesting create read pulumi:pulumi:StackReference dev.infra.infratesting-reference
+  pulumi:providers:kubernetes clusterProvider create
+  kubernetes:apps/v1:DaemonSet default/keyVaultCSI-secrets-store-csi-driver create
+  kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition secretproviderclasspodstatuses.secrets-store.csi.x-k8s.io create
+  kubernetes:storage.k8s.io/v1:CSIDriver secrets-store.csi.k8s.io create
+  kubernetes:rbac.authorization.k8s.io/v1:ClusterRole secretprovidersyncing-role create
+  kubernetes:rbac.authorization.k8s.io/v1:ClusterRole secretproviderclasses-role create
+  kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding secretprovidersyncing-rolebinding create
+  kubernetes:rbac.authorization.k8s.io/v1:ClusterRoleBinding secretproviderclasses-rolebinding create
+  kubernetes:core/v1:ServiceAccount default/csi-secrets-store-provider-azure create
+  kubernetes:core/v1:ServiceAccount default/secrets-store-csi-driver create
+  kubernetes:apiextensions.k8s.io/v1:CustomResourceDefinition secretproviderclasses.secrets-store.csi.x-k8s.io create
+  kubernetes:apps/v1:DaemonSet default/keyVaultCSI-csi-secrets-store-provider-azure create
+  pulumi:pulumi:Stack baseline-k8s-dev.k8s.infratesting create

Resources:

Context (Environment)

Run with pulumi 2.24.1, pulumi-kubernetes 2.8.4, using pulumi/x/automation.

Could also be linked to https://github.com/pulumi/pulumi-kubernetes/issues/1505

Affected feature

MatteoCalabro-TomTom commented 3 years ago

1505 -> https://github.com/helm/helm/issues/9533

gregmuellegger commented 2 years ago

I've run into this problem as well. It was frustrating because I anticipated to not have to install helm for making pulumi work with helm charts as it states that it bundles the helm go implementation directly.

Turns out that I had old files of a repository lying around. Seems like does work once I remove the old repository using helm repo remove ...

gitfool commented 2 years ago

I'm also hitting this every now and then. As above I presume the helm cli is not actually needed by pulumi, but it's installed for other reasons, so it would be great if pulumi could somehow bypass requiring a helm repo update...

mikeseese commented 1 year ago

This is still a helm issue (I created a new issue https://github.com/helm/helm/issues/11961) to better document what's going on.

Generally speaking, helm is failing because it's expecting there to be a cache even if the command is targeting a remote repo/chart; that cache gets cleared periodically (at least on Windows) causing helm to freak out even though it shouldn't need the cache anyway.

A workaround is to just run helm repo update to make sure the cache gets filled again of whatever local repos you have. If it makes sense for you, you can also remove the local helm repos so it doesn't think there's a missing cache.

EronWright commented 3 months ago

Thanks again @mikeseese for the analysis and efforts to fix the upstream issue (https://github.com/helm/helm/issues/11961). Helm assumes that the repo cache is valid for all added repositories, even if the installation task doesn't rely on those repositories.

EronWright commented 1 month ago

TLDR; this happens when you've added a repository to Helm (helm repo add) and then cleared the repository cache. Helm doesn't automatically rebuild the cache.

The workaround is to a) remove the repo (helm repo remove REPO-NAME) or b) rebuild the cache (helm repo update).

An easy repro for this issue is to customize the cache location, e.g.

$ helm repo add nginx https://helm.nginx.com/stable
$ mkdir ./helm
$ export HELM_REPOSITORY_CACHE=./helm/repository
$ helm template --repo https://helm.nginx.com/stable nginx-ingress
Error: no cached repo found. (try 'helm repo update'): open helm/repository/nginx-index.yaml: no such file or directory

It is apparent that the template command doesn't automatically do helm repo update, it simply assumes the cache is good. That's true only for the main chart, because it does automatically update the repo cache as necessary while resolving the chart's dependencies. https://github.com/helm/helm/pull/11963 allows Helm to skip over bad cache entries, but doesn't seek to solve the broader issue.

To be clear, this issue manifests itself only when the repo URL is found in the set of locally-configured repositories. If we set HELM_REPOSITORY_CONFIG to an empty file, Helm appears to download and cache automatically as expected. This means that the "adhoc repo" use-case works today as expected.

$ mkdir ./helm
$ touch ./helm/repositories.yaml
$ export HELM_REPOSITORY_CONFIG=./helm/repositories.yaml
$ export HELM_REPOSITORY_CACHE=./helm/repository
$ helm template --repo https://helm.nginx.com/stable nginx-ingress
---
# Source: nginx-ingress/templates/controller-serviceaccount.yaml
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
  name: nginx
...