search

roboll / helmfile

Deploy Kubernetes Helm Charts
MIT License
4.04k stars 565 forks source link

Confusing error message on chart verification error #467

Open andrewnazarov opened 5 years ago

andrewnazarov commented 5 years ago

I'm facing the following issue. I defined a bunch of external services in my helmfile. Then I tried to sync everything. Doing helm sync I got

Upgrading stable/cert-manager Upgrading gitlab/gitlab-runner Upgrading stable/minio Upgrading stable/prometheus-operator Upgrading stable/chartmuseum Error: failed to download "gitlab/gitlab-runner" (hint: running helm repo update may help)

Error: failed to download "stable/prometheus-operator" (hint: running helm repo update may help)

Error: failed to download "stable/cert-manager" (hint: running helm repo update may help)

Error: failed to download "stable/chartmuseum" (hint: running helm repo update may help)

Error: failed to download "stable/minio" (hint: running helm repo update may help)

err: release "runner" in "helmfile.yaml" failed: exit status 1 err: release "prom" in "helmfile.yaml" failed: exit status 1 err: release "cert-manager" in "helmfile.yaml" failed: exit status 1 err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1 err: release "s3" in "helmfile.yaml" failed: exit status 1 exit status 1

What it's all about? In docs it's stated that helmfile sync updates chart repositories and dependencies under the hood.

Btw, I have another helmfile.yaml located in a different folder, it has a mix of internal (with local charts) and external services. helmfile sync works ok inside this folder.

helmfile version v0.43.2

mumoshu commented 5 years ago

Hey! Thanks for the report.

Would you mind rerunning it with helmfile --log-level=debug and share me the result? Perhaps that helps investigating the underlying issue. 2019年2月6日(水) 20:03 Andrew Nazarov notifications@github.com:

I'm facing the following issue. I defined a bunch of external services in my helmfile. Then I tried to sync everything. Doing helm sync I got

Upgrading stable/cert-manager Upgrading gitlab/gitlab-runner Upgrading stable/minio Upgrading stable/prometheus-operator Upgrading stable/chartmuseum Error: failed to download "gitlab/gitlab-runner" (hint: running helm repo update may help)

Error: failed to download "stable/prometheus-operator" (hint: running helm repo update may help)

Error: failed to download "stable/cert-manager" (hint: running helm repo update may help)

Error: failed to download "stable/chartmuseum" (hint: running helm repo update may help)

Error: failed to download "stable/minio" (hint: running helm repo update may help)

err: release "runner" in "helmfile.yaml" failed: exit status 1 err: release "prom" in "helmfile.yaml" failed: exit status 1 err: release "cert-manager" in "helmfile.yaml" failed: exit status 1 err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1 err: release "s3" in "helmfile.yaml" failed: exit status 1 exit status 1

What it's all about? In docs it's stated that helmfile sync updates chart repositories and dependencies under the hood.

Btw, I have another helmfile.yaml located in a different folder, it has a mix of internal (with local charts) and external services. helmfile sync works ok inside this folder.

helmfile version v0.43.2

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/roboll/helmfile/issues/467, or mute the thread https://github.com/notifications/unsubscribe-auth/AABV-b_D_ATloI6l0wsYZVF-jxhWfujhks5vKraOgaJpZM4alABX .

andrewnazarov commented 5 years ago

Sure. Here it is (sensitive info is removed):

processing file "helmfile.yaml" in directory "."
second-pass rendering result of "helmfile.yaml":
 0: repositories:
 1:   - name: gitlab
 2:     url: https://charts.gitlab.io
 3:   - name: stable
 4:     url: https://kubernetes-charts.storage.googleapis.com
 5: 
 6: helmDefaults:           
 7:   tillerNamespace: gitlab-managed-apps
 8:   kubeContext: gke_xxxx-XXXX_europe-west3-a_cluster-xxxx
 9:   verify: true
10:   wait: true
11:   timeout: 600
12:   recreatePods: false
13:   force: false
14: 
15: releases:
16:   - name: runner
17:     namespace: gitlab-managed-apps
18:     chart: gitlab/gitlab-runner
19:     version: ~0.1.45
20:     values:
21:     - "./gitlab-runner/gitlab-runner-values.yml"
22: 
23:   - name: prom
24:     namespace: monitoring
25:     chart: stable/prometheus-operator
26:     version: ~2.1.1
27:     values:
28:     - "./prometheus/values.yml"
29: 
30:   - name: s3
31:     namespace: gitlab-managed-apps
32:     chart: stable/minio
33:     version: ~2.4.1
34:     values:
35:     - "./minio/minio-values.yml"
36: 
37:   - name: cert-manager
38:     namespace: default
39:     chart: stable/cert-manager
40:     version: ~0.5.0
41:     set:
42:       - name: rbac.create
43:         value: false
44: 
45:   - name: chartmuseum
46:     namespace: gitlab-managed-apps
47:     chart: stable/chartmuseum
48:     version: ~1.9.0
49:     set:
50:       - name: env.open.STORAGE
51:         value: local
52:       - name: persistence.enabled
53:         value: true
54:       - name: persistence.accessMode
55:         value: ReadWriteOnce
56:       - name: persistence.size
57:         value: 10Gi

Adding repo gitlab https://charts.gitlab.io
exec: helm repo add gitlab https://charts.gitlab.io
"gitlab" has been added to your repositories

Adding repo stable https://kubernetes-charts.storage.googleapis.com
exec: helm repo add stable https://kubernetes-charts.storage.googleapis.com
"stable" has been added to your repositories

Updating repo
exec: helm repo update
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "gitlab" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 

worker 1/5 started
worker 5/5 started
worker 3/5 started
worker 2/5 started
worker 4/5 started
worker 4/5 finished
worker 2/5 finished
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx-4/infra/init/prometheus/values.yml. produced:
coreDns:
  enabled: false

kubeDns:
  enabled: true

alertmanager:
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

prometheus:
  prometheusSpec:
    storage:
      volumeClaimTemplate:
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

grafana:
  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: "nginx"
      certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    hosts:
      - grafana.test.itc-engineering.com
    tls:
      - secretName: grafana-myingress-cert
        hosts:
          - grafana.test.itc-engineering.com
  persistence:
    enabled: true
    accessModes: ["ReadWriteOnce"]
    size: 10Gi
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx/infra/init/gitlab-runner/gitlab-runner-values.yml. produced:
# This file only contains subset of available configuration, see https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml

## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
##
gitlabUrl: https://gitlab.itc-engineering.com

## The Registration Token for adding new Runners to the GitLab Server. This must
## be retreived from your GitLab Instance.
## ref: https://docs.gitlab.com/ce/ci/runners/README.html#creating-and-registering-a-runner
##
runnerRegistrationToken: "XXXXXXXXX"

## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-unregister
##
unregisterRunners: false

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 3

## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 15

## Configuration for the Pods that that the runner launches for each new job
##
runners:

  ## Run all containers with the privileged flag enabled
  ## This will allow the docker:dind image to run if you need to run Docker
  ## commands. Please read the docs before turning this on:
  ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
  ##
  privileged: true

  ## Distributed runners caching
  ## ref: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/autoscale.md#distributed-runners-caching
  ##
  ## Create a secret 's3access' containing 'accesskey' & 'secretkey'
  ## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
  ##
  ## $ kubectl create secret generic s3access --\
  ##   --from-literal=accesskey="YourAccessKey" \
  ##   --from-literal=secretkey="YourSecretKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  cache:
    cacheType: s3
    s3ServerAddress: s3-minio:9000
    s3BucketName: cache
    s3CacheInsecure: true
    cacheShared: true

  ## Build Container specific configuration
  ##
  builds: 
    memoryLimit: 10Gi
    cpuRequests: 1000m
    memoryRequests: 8Gi

  ## Service Container specific configuration
  ##
  services: {}
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    # cpuRequests: 100m
    # memoryRequests: 128Mi

  ## Helper Container specific configuration
  ##
  helpers: {}
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    # cpuRequests: 100m
    # memoryRequests: 128Mi
    # image: gitlab/gitlab-runner-helper:x86_64-latest

worker 5/5 finished
successfully generated the value file at /Users/andrewnazarov/Repo/xxxx-4/infra/init/minio/minio-values.yml. produced:
# This file only contains subset of available configuration, see https://github.com/kubernetes/charts/blob/master/stable/minio/values.yaml

## Set default accesskey, secretkey
accessKey: "XXXXXXX"
secretKey: "XXXXXXX"

## Create a bucket after minio install
##
defaultBucket:
  enabled: true
  ## If enabled, must be a string with length > 0
  name: cache
  ## Can be one of none|download|upload|public
  policy: none
  ## Purge if bucket exists already
  purge: false

worker 3/5 finished
worker 1/5 finished
worker 5/5 started
worker 3/5 started
worker 4/5 started
Upgrading stable/cert-manager
Upgrading stable/prometheus-operator
Upgrading stable/chartmuseum
exec: helm upgrade --install --reset-values cert-manager stable/cert-manager --version ~0.5.0 --verify --wait --timeout 600 --namespace default --set rbac.create=false --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
exec: helm upgrade --install --reset-values prom stable/prometheus-operator --version ~2.1.1 --verify --wait --timeout 600 --namespace monitoring --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values637418716 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
exec: helm upgrade --install --reset-values chartmuseum stable/chartmuseum --version ~1.9.0 --verify --wait --timeout 600 --namespace gitlab-managed-apps --set env.open.STORAGE=local --set persistence.enabled=true --set persistence.accessMode=ReadWriteOnce --set persistence.size=10Gi --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
worker 1/5 started
worker 2/5 started
Upgrading stable/minio
exec: helm upgrade --install --reset-values s3 stable/minio --version ~2.4.1 --verify --wait --timeout 600 --namespace gitlab-managed-apps --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values565451374 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
Upgrading gitlab/gitlab-runner
exec: helm upgrade --install --reset-values runner gitlab/gitlab-runner --version ~0.1.45 --verify --wait --timeout 600 --namespace gitlab-managed-apps --values /var/folders/pl/k4wqlgfj3cjf6b2p7qblq1200000gn/T/values120155275 --tiller-namespace=gitlab-managed-apps --kube-context=gke_xxxx-XXXXXX_europe-west3-a_cluster-xxxx
Error: failed to download "gitlab/gitlab-runner" (hint: running `helm repo update` may help)

worker 2/5 finished
Error: failed to download "stable/cert-manager" (hint: running `helm repo update` may help)

worker 3/5 finished
Error: failed to download "stable/minio" (hint: running `helm repo update` may help)

worker 1/5 finished
Error: failed to download "stable/prometheus-operator" (hint: running `helm repo update` may help)

worker 4/5 finished
Error: failed to download "stable/chartmuseum" (hint: running `helm repo update` may help)

worker 5/5 finished
err: release "runner" in "helmfile.yaml" failed: exit status 1
err: release "cert-manager" in "helmfile.yaml" failed: exit status 1
err: release "s3" in "helmfile.yaml" failed: exit status 1
err: release "prom" in "helmfile.yaml" failed: exit status 1
err: release "chartmuseum" in "helmfile.yaml" failed: exit status 1
exit status 1
andrewnazarov commented 5 years ago

My first assumption is probably it has something to do with the version of one of the releases. For example, the version of cert-manager is pinned not to the most recent.

andrewnazarov commented 5 years ago

My first assumption is probably it has something to do with the version of one of the releases. For example, the version of cert-manager is pinned not to the most recent.

No, I've just checked it out. Versions are not the case.

andrewnazarov commented 5 years ago

I found it. helmDefaults.verify=true causes the issue. I set it to false and helmfile sync started working.

By the way, it seems for some reason -i flag doesn't work with --log-level=debug. I'll double check that.

mumoshu commented 5 years ago

Thanks for your effort!

I found it. helmDefaults.verify=true causes the issue. I set it to false and helmfile sync started working.

Interesting. All it does should be just adding --verify to helm commands called by helmfile. Do you have any idea how it resulted in such errors...?

By the way, it seems for some reason -i flag doesn't work with --log-level=debug. I'll double check that.

How the whole set of flags are ordered in your command? I thought both --log-level and -i flags must precede before sync so that the command looks like helm --log-level=debug -i sync.

andrewnazarov commented 5 years ago

Interesting. All it does should be just adding --verify to helm commands called by helmfile. Do you have any idea how it resulted in such errors...?

Probably it has somethng to do with:

helm upgrade --dry-run --debug --install --reset-values chartmuseum stable/chartmuseum --version ~1.9.0 --verify --wait --timeout 600 --namespace gitlab-managed-apps --set env.open.STORAGE=local --set persistence.enabled=true --set persistence.accessMode=ReadWriteOnce --set persistence.size=10Gi --tiller-namespace=gitlab-managed-apps --kube-context=gke_iris-206888_europe-west3-a_cluster-iris
[debug] Created tunnel using local port: '60206'

[debug] SERVER: "127.0.0.1:60206"

Error: Failed to fetch provenance "https://kubernetes-charts.storage.googleapis.com/chartmuseum-1.9.0.tgz.prov"

In that case the error message Error: failed to download "stable/chartmuseum" (hint: running 'helm repo update' may help) is misleading.

andrewnazarov commented 5 years ago

How the whole set of flags are ordered in your command? I thought both --log-level and -i flags must precede before sync so that the command looks like helm --log-level=debug -i sync

The order of flags is right.

history | grep -E 'helmfile.*sync'
  752  helmfile --log-level=debug -i sync
  753  helmfile --log-level=debug -i sync
  754  helmfile -i sync
  756  helmfile -i sync
  757  helmfile -i sync
  761  helmfile -i sync
  767  helmfile -i sync
  768  helmfile -i sync
  777  helmfile -i sync
  778  helmfile -i sync
  779  helmfile --log-level=debug -i sync

But it seems that helmfile sync doesn't respect -i flag at all, even without --log-level flag. I don't want to create a new issue at the moment as there are too much noise and false -positives coming from me)).

mumoshu commented 5 years ago

@andrewnazarov Thanks a lot for your support!

Agreed that helmfile should somehow notice the user about the verification error.

Regarding -i, yes, sorry about that! We had a dedicated issue NOT to accept -i for unsupported helmfile commands #416.

Xtigyro commented 4 years ago

Yesterday with helmfile v0.130 I started seeing the same problem. Still need to validate it completely whether it's the exact same root cause.

4c74356b41 commented 4 years ago

@mumoshu just hit this same error, started yesterday as well, same fix helped.

mumoshu commented 4 years ago

@Xtigyro @4c74356b41 On which charts did you start seeing the error? (I doubt if it has something to do with the deprecation of stable and incubator chart repositories

4c74356b41 commented 4 years ago

with stable, yeah, but helm pull stable/grafana works fine, so it looks like you are correct here.

worldofgeese commented 3 years ago

Just noticed the same error myself using the bitnami/nginx-ingress-controller chart and verify=true. This is using the latest helmfile release.

ryanpeach commented 3 years ago

Same for jenkins/jenkins and verify=true

woodcockjosh commented 3 years ago

failing for bitnami/external-dns

hungnp-3073 commented 3 years ago

Same issue for eks/aws-load-balancer-controller when verify=true.

abvijaykumar commented 2 years ago

I am getting this issue with almost all valid helm charts, which work fine with helm command, looks like this is a bug. and sometimes it works, sometimes it fails.

mumoshu commented 2 years ago

verify works only on signed charts. Have you really tried it with --verify when it didn't reproduce for you with the vanilla helm? AFAIK, charts are rarely signed.

abvijaykumar commented 2 years ago

Yes I tried verify. Jt still fails. Another behaviours I see is that. Sometimes it goes thru with one chart and fails with another chart. And it's random.

mumoshu commented 2 years ago

@worldofgeese bitnami/nginx-ingress-controller seems not signed:

$ helm repo add bitnami https://charts.bitnami.com/bitnami --force-update
"bitnami" has been added to your repositories
$ helm template my-release bitnami/nginx-ingress-controller --verify
Error: failed to download "bitnami/nginx-ingress-controller"

@ryanpeachjenkins/jenkins seems not signed:

$ helm repo add jenkins https://charts.jenkins.io --force-update
"jenkins" has been added to your repositories
$ helm template my-release jenkins/jenkins --verify
Error: failed to download "jenkins/jenkins"

@hungnp-3073 eks/aws-load-balancer-controller seems not signed:

$ helm repo add eks https://aws.github.io/eks-charts
"eks" has been added to your repositories
$ helm template my-release eks/aws-load-balancer-controller --verify
Error: failed to download "eks/aws-load-balancer-controller"

Everyone, check your chart and let me know which chart doesn't work only with helmfile.

abvijaykumar commented 2 years ago

HI

@worldofgeese bitnami/nginx-ingress-controller seems not signed:

$ helm repo add bitnami https://charts.bitnami.com/bitnami --force-update
"bitnami" has been added to your repositories
$ helm template my-release bitnami/nginx-ingress-controller --verify
Error: failed to download "bitnami/nginx-ingress-controller"

@ryanpeachjenkins/jenkins seems not signed:

$ helm repo add jenkins https://charts.jenkins.io --force-update
"jenkins" has been added to your repositories
$ helm template my-release jenkins/jenkins --verify
Error: failed to download "jenkins/jenkins"

@hungnp-3073 eks/aws-load-balancer-controller seems not signed:

$ helm repo add eks https://aws.github.io/eks-charts
"eks" has been added to your repositories
$ helm template my-release eks/aws-load-balancer-controller --verify
Error: failed to download "eks/aws-load-balancer-controller"

Everyone, check your chart and let me know which chart doesn't work only with helmfile.

How to check if its signed or not. I am using all stable charts..and these errors are random, sometimes they go thru, sometimes they fail.

for example it failed with the following error Error: failed to download "prometheus-community/prometheus-postgres-exporter"

This was working yesterday

mumoshu commented 2 years ago

@abvijaykumar Honestly speaking I've never tried to struggle with --verify for community charts as they are mostly unsigned. But anyway- the easiest way I can think of is to run the helm command with and without the --verify option. If it works only without --verify, it's unsigned.

abvijaykumar commented 2 years ago

@abvijaykumar Honestly speaking I've never tried to struggle with --verify for community charts as they are mostly unsigned. But anyway- the easiest way I can think of is to run the helm command with and without the --verify option. If it works only without --verify, it's unsigned.

Here is what my current yaml looks like. DO u see anything wrong.

repositories:

helmDefaults: verify: false wait: true waitForJobs: true createNamespace: true

commonLabels: application-name: bozo-book-library

releases:

mumoshu commented 2 years ago

@abvijaykumar Looks good to me. If it doesn't work, perhaps any of your chart repositories are not working correctly?

abvijaykumar commented 2 years ago

@abvijaykumar Looks good to me. If it doesn't work, perhaps any of your chart repositories are not working correctly?

Its actually random, sometimes some repositories go through, sometimes the same repo fails. Its very unpredictable, so I am not able to code this into my GitOps. Planning to move to Terraform