Helm fails to download chart when installing but claims deployment worked

[WolfSkin0]

I've been trying to deploy cert-manager via helm to a RKE kubernetes cluster both locally and in the cloud. However, every time Helm claims that the chart is deployed and k8s claims it isn't. Using --wait/--atomic has merely ended in timeouts and --verify has claimed that helm can't fetch the chart at all (even though using helm to download the chart and untar it works fine as long as I don't try to install).

wolfskin@Arch:~/repos/rke|⇒  helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "rancher-latest" chart repository
...Successfully got an update from the "jetstack" chart repository
Update Complete. ⎈Happy Helming!⎈
wolfskin@Arch:~/repos/rke|⇒  kubectl create namespace cattle-system
namespace/cattle-system created
wolfskin@Arch:~/repos/rke|⇒  kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.0.4/cert-manager.crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
wolfskin@Arch:~/repos/rke|⇒  kubectl create namespace cert-manager
namespace/cert-manager created
wolfskin@Arch:~/repos/rke|⇒  helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --version v1.0.4 \
> --verify \
> --atomic
Error: failed to download "jetstack/cert-manager" at version "v1.0.4" (hint: running `helm repo update` may help)
wolfskin@Arch:~/repos/rke|⇒

Any advice that you could give me on this would be much appreciated as I'm completely stumped and under pressure to get this working.

Output of helm version:

version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.15.6"}

Output of kubectl version:

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.4", GitCommit:"d360454c9bcd1634cf4cc52d1867af5491dc9c5f", GitTreeState:"archive", BuildDate:"2020-11-25T13:19:56Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.14", GitCommit:"f238f5142728be4033c37aa0ad69bf806090beae", GitTreeState:"clean", BuildDate:"2020-11-11T13:03:54Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Cloud Provider/Platform (AKS, GKE, Minikube etc.): Hetzner VPS + RKE

[mattfarina]

Can you run the command helm search repo jetstack/cert-manager -l? This will help you see of 1.0.4 is an available version.

[WolfSkin0]

Here is the output:

wolfskin@Arch:~|⇒  helm search repo jetstack/cert-manager -l
NAME                    CHART VERSION   APP VERSION DESCRIPTION
jetstack/cert-manager   v1.1.0          v1.1.0      A Helm chart for cert-manager
jetstack/cert-manager   v1.0.4          v1.0.4      A Helm chart for cert-manager
jetstack/cert-manager   v1.0.3          v1.0.3      A Helm chart for cert-manager
jetstack/cert-manager   v1.0.2          v1.0.2      A Helm chart for cert-manager
jetstack/cert-manager   v1.0.1          v1.0.1      A Helm chart for cert-manager
jetstack/cert-manager   v1.0.0          v1.0.0      A Helm chart for cert-manager
jetstack/cert-manager   v0.16.1         v0.16.1     A Helm chart for cert-manager
jetstack/cert-manager   v0.16.0         v0.16.0     A Helm chart for cert-manager
jetstack/cert-manager   v0.15.2         v0.15.2     A Helm chart for cert-manager
jetstack/cert-manager   v0.15.1         v0.15.1     A Helm chart for cert-manager
jetstack/cert-manager   v0.15.0         v0.15.0     A Helm chart for cert-manager
jetstack/cert-manager   v0.14.3         v0.14.3     A Helm chart for cert-manager
jetstack/cert-manager   v0.14.2         v0.14.2     A Helm chart for cert-manager
jetstack/cert-manager   v0.14.1         v0.14.1     A Helm chart for cert-manager
jetstack/cert-manager   v0.14.0         v0.14.0     A Helm chart for cert-manager
jetstack/cert-manager   v0.13.1         v0.13.1     A Helm chart for cert-manager
jetstack/cert-manager   v0.13.0         v0.13.0     A Helm chart for cert-manager
jetstack/cert-manager   v0.12.0         v0.12.0     A Helm chart for cert-manager
jetstack/cert-manager   v0.11.1         v0.11.1     A Helm chart for cert-manager
jetstack/cert-manager   v0.11.0         v0.11.0     A Helm chart for cert-manager
jetstack/cert-manager   v0.10.1         v0.10.1     A Helm chart for cert-manager
jetstack/cert-manager   v0.10.0         v0.10.0     A Helm chart for cert-manager
jetstack/cert-manager   v0.9.1          v0.9.1      A Helm chart for cert-manager
jetstack/cert-manager   v0.9.0          v0.9.0      A Helm chart for cert-manager
jetstack/cert-manager   v0.8.1          v0.8.1      A Helm chart for cert-manager
jetstack/cert-manager   v0.8.0          v0.8.0      A Helm chart for cert-manager
jetstack/cert-manager   v0.7.2          v0.7.2      A Helm chart for cert-manager
jetstack/cert-manager   v0.7.1          v0.7.1      A Helm chart for cert-manager
jetstack/cert-manager   v0.7.0          v0.7.0      A Helm chart for cert-manager
jetstack/cert-manager   v0.6.0          v0.6.0      A Helm chart for cert-manager
jetstack/cert-manager   v0.5.2          v0.5.2      A Helm chart for cert-manager
wolfskin@Arch:~|⇒

The version is definitely there, and I get the exact same issue with v1.1.0

[mattfarina]

Are you able to run helm pull jetstack/cert-manager --version v1.0.4? This should download the chart. I ask this because the index with the information used to know about a version and have metadata for search is separate from the assets. If the asset can't be reached it's a second issue.

If that works, can you use the --debug flag on the helm install command to get additional detail. This flag works on any Helm command to get additional information.

[WolfSkin0]

Ok, so I can run the first command and I can successfully pull down the tar.gz file. When I run helm install with debug, I get the following: https://termbin.com/6dwy I have also included some other command outputs in there pertaining to showcasing that the install still fails for some reason. I've also figured out that the verify flag is causing problems because there is not provenance file. However its removal seems to have no effect on the actual deployment problem as seen in the output above

[WolfSkin0]

So that seems to have at least solved the download issue

[github-actions[bot]]

This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.