helm repo add bitnami gives not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

[chukka]

Which chart: Not chart but bitnami helm repo - helm repo add bitnami https://charts.bitnami.com/bitnami The name (and version) of the affected chart

Describe the bug As part of CI tests, we add the bitnami repo and do some chart-testing. This breaks CI flow. is there any option to retry or is it a known issue

To Reproduce Steps to reproduce the behavior:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

Expected behavior A clear and concise description of what you expected to happen.

Version of Helm and Kubernetes: 3.5.4

Additional context This doesn't happen always but fails 2 out of 10 times atleast

[carrodher]

I was not able to reproduce the issue, I tried several times and it worked fine for all of them. Maybe it's related to the AWS outage that happened yesterday, see some news https://www.theguardian.com/technology/2021/dec/15/amazon-down-web-services-outage-netflix-slack-ring-doordash-latest

Are you still facing the issue?

[chukka]

@carrodher This issue happens intermittently , This is not the first time it happened for us yesterday .

[carrodher]

Hi @chukka, could you tell us since when this problem occurred in your case? My 2cts is that it can be related to any connectivity issue produced by a third-party outage like AWS, DNS provider, etc. We have several automations to release the charts that are constantly adding the repository and I didn't see any issue, or if there were an issue it was transient and solved with the automatic retries.

[GCHQDeveloper926]

I've had this happen 2-3 times on 15 December and 3 times this morning 17 December:

Downloading common from repo https://charts.bitnami.com/bitnami
Save error occurred:  could not find : chart rabbitmq not found in https://charts.bitnami.com/bitnami: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR
Deleting newly downloaded charts, restoring pre-update state
Error: could not find : chart rabbitmq not found in https://charts.bitnami.com/bitnami: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

[javsalgar]

Hi,

In my case I wasn't able to reproduce the issue either.

[andybjourney]

I'm seeing this as well, and it is intermittent. I got this from running terraform locally against a k3d cluster via the helm provider:

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

[javsalgar]

I forwarded this to the engineering team so they can take a look.

[tebaly]

Doesn't work now. Already about 30 minutes

ArgoCD

rpc error: code = Unknown desc = Manifest generation error (cached): helm repo add https://charts.bitnami.com/bitnami https://charts.bitnami.com/bitnami failed exit status 1: Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR

[carrodher]

Thanks for letting us know, we are looking into it

[carrodher]

Thanks to everyone for reporting this issue. We are facing some transient issues with the Helm registry, it should work if you retry it:

$ helm repo add bitnami2 https://charts.bitnami.com/bitnami
"bitnami2" has been added to your repositories

About users trying to access using the browser, please note this URL is designed for work with the Helm CLI, the only path accessible via browser is https://charts.bitnami.com/bitnami/index.yaml where you can download the index.yaml

We will update the thread with more information once properly analyzed the issue on our side. Thanks!

[carrodher]

We have been doing some tests, for example the following script was running during a couple of days:

#!/bin/bash

set -e

export HELM_DEBUG=1

error() {
    echo "See file /tmp/chart.$$.log for debugging info"
}

command -v helm || exit 1

mkdir -p /tmp/charts/

# Do not fail here
helm repo remove bitnami || true

(helm repo add bitnami https://charts.bitnami.com/bitnami 2> /tmp/charts/$$.log) || error

From the original 20063 probes, about 9.1% failed with this message:

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peerError: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peerhelm.go:88: [debug] stream error: stream ID 1; INTERNAL_ERROR; received from peerlooks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reachedmain.(*repoAddOptions).run helm.sh/helm/v3/cmd/helm/repo_add.go:206main.newRepoAddCmd.func1 helm.sh/helm/v3/cmd/helm/repo_add.go:82github.com/spf13/cobra.(*Command).execute github.com/spf13/cobra@v1.2.1/command.go:856github.com/spf13/cobra.(*Command).ExecuteC github.com/spf13/cobra@v1.2.1/command.go:974github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.2.1/command.go:902main.main helm.sh/helm/v3/cmd/helm/helm.go:87runtime.main runtime/proc.go:255runtime.goexit runtime/asm_amd64.s:1581

This error seems to be caused by a network issue, it can be potentially caused on both sides: client or server: is not a valid chart repository or cannot be reached

Line https://github.com/helm/helm/blob/39ca699ca790e02ba36753dec6ba4177cc68d417/cmd/helm/repo_add.go#L205 points to be a client error, but it's not a determinant clue.

We have performed some tests and we can fetch the file https://charts.bitnami.com/bitnami/index.yaml 100% of the cases, but sometimes helm fails after curl succeeding:

# Curl 
*   Trying 99.84.160.70...*   Trying 99.84.160.70...* TCP_NODELAY set* Connected to charts.bitnami.com (99.84.160.70) port 443 (#0)* ALPN, offering h2* ALPN, offering http/1.1* successfully set certificate verify locations:*   CAfile: /etc/ssl/cert.pem  CApath: none* TLSv1.2 (OUT), TLS handshake, Client hello (1):} [232 bytes data]* TLSv1.2 (IN), TLS handshake, Server hello (2):{ [100 bytes data]* TLSv1.2 (IN), TLS handshake, Certificate (11):{ [4820 bytes data]* TLSv1.2 (IN), TLS handshake, Server key exchange (12):{ [300 bytes data]* TLSv1.2 (IN), TLS handshake, Server finished (14):{ [4 bytes data]* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):} [37 bytes data]* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):} [1 bytes data]* TLSv1.2 (OUT), TLS handshake, Finished (20):} [16 bytes data]* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):{ [1 bytes data]* TLSv1.2 (IN), TLS handshake, Finished (20):{ [16 bytes data]* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256* ALPN, server accepted to use h2* Server certificate:*  subject: CN=charts.bitnami.com*  start date: Feb 14 00:00:00 2021 GMT*  expire date: Mar 15 23:59:59 2022 GMT*  subjectAltName: host "charts.bitnami.com" matched cert's "charts.bitnami.com"*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon*  SSL certificate verify ok.* Using HTTP2, server supports multi-use* Connection state changed (HTTP/2 confirmed)* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0* Using Stream ID: 1 (easy handle 0x7ffbbb00d600)> GET /bitnami/index.yaml HTTP/2> Host: charts.bitnami.com> User-Agent: curl/7.64.1> Accept: */*> * Connection state changed (MAX_CONCURRENT_STREAMS == 128)!< HTTP/2 200 < content-type: binary/octet-stream< content-length: 11287899< last-modified: Tue, 11 Jan 2022 14:36:14 GMT< x-amz-server-side-encryption: AES256< x-amz-version-id: nmDdVr8LQPExrl5WNUdeu5redpWHeUxw< accept-ranges: bytes< server: AmazonS3< date: Tue, 11 Jan 2022 14:52:52 GMT< etag: "04935efc90d38c851fb4009f3b700563"< x-cache: RefreshHit from cloudfront< via: 1.1 7fa75077148a5e108de410272ea8ccc4.cloudfront.net (CloudFront)< x-amz-cf-pop: ORD52-C2< x-amz-cf-id: oWfuxcKqMqT0AP-kpwCfGBX-IA42jtR6mjFoXGazr0-p2GmBIoFwfQ==< { [15983 bytes data]* Connection #0 to host charts.bitnami.com left intact* Closing connection 0

# Helm
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peerhelm.go:88: [debug] stream error: stream ID 1; INTERNAL_ERROR; received from peerlooks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reachedmain.(*repoAddOptions).run helm.sh/helm/v3/cmd/helm/repo_add.go:206main.newRepoAddCmd.func1 helm.sh/helm/v3/cmd/helm/repo_add.go:82github.com/spf13/cobra.(*Command).execute github.com/spf13/cobra@v1.2.1/command.go:856github.com/spf13/cobra.(*Command).ExecuteC github.com/spf13/cobra@v1.2.1/command.go:974github.com/spf13/cobra.(*Command).Execute github.com/spf13/cobra@v1.2.1/command.go:902main.main helm.sh/helm/v3/cmd/helm/helm.go:87runtime.main runtime/proc.go:255runtime.goexit runtime/asm_amd64.s:1581

[JoepWeijers]

I'm also seeing this issue pop up in a CI build that calls Helm about once per day.

[viniciusfacco]

I'm having the same issue. This is the message I get:

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://charts.bitnami.com/bitnami/index.yaml": read tcp 10.163.207.67:61516->108.138.167.54:443: wsarecv: An existing connection was forcibly closed by the remote host.

[Segamegadrive]

Hi, I'm also facing a similar type of error. Jenkins build fails and I'm using Terraform HEML Provider to deploy Prometheus HEML chat from bitnami. Tried to get around with it by creating a docker image as well; defining commands to install HELM 3 and adding CMD to add repos, and still didn't help. I also tried replacing the TF HELM repository attribute with "/.cache/helm/repository/" but throws a Protocol Handler error.

Below is the error I have been getting:

module.prometheus.helm_release.grafana: Creating...
module.prometheus.helm_release.prometheus: Creating...
module.prometheus.helm_release.grafana: Still creating... [10s elapsed]
module.prometheus.helm_release.prometheus: Still creating... [10s elapsed]

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: open /.cache/helm/repository/HpOj5uooN5ifybCQee2mfuZUHA4=-index.yaml: no such file or directory

  with module.prometheus.helm_release.prometheus,
  on .terraform/modules/prometheus/main.tf line 16, in resource "helm_release" "prometheus":
  16: resource "helm_release" "prometheus" {

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: open /.cache/helm/repository/ON43pZ61a8pJuVDiqDkXwhDeE0I=-index.yaml: no such file or directory

  with module.prometheus.helm_release.grafana,
  on .terraform/modules/prometheus/main.tf line 25, in resource "helm_release" "grafana":
  25: resource "helm_release" "grafana" {

My Terraform HEML module below, tried with chart name without bitnami/ path too and still throws same error as above:

resource "helm_release" "prometheus" {
  name             = "prometheus"
  repository       = "https://charts.bitnami.com/bitnami"
  chart            = "bitnami/prometheus"
  namespace        = "monitoring"
  create_namespace = true
}

// Grafana HEML chart for Visualisation of telemetry data
resource "helm_release" "grafana" {
  name             = "grafana"
  repository       = "https://charts.bitnami.com/bitnami"
  chart            = "bitnami/grafana"
  namespace        = "monitoring"
  create_namespace = true
}

Any info and advice would be grateful, please. Thank you.

[carrodher]

Hi, monitoring didn't show bad connections or alike, on our side, are you still facing this intermittent issue?

[snorlaX-sleeps]

We are also using Terraform to deploy the external DNS Helm chart and have experienced it several times today. It did not however occur with the metrics-server Helm Chart.

[JoepWeijers]

Yes, we are still seeing this issue

[DanielJRusev]

https://charts.bitnami.com/bitnami

AccessDenied Access Denied SN5Q4NA1V71A1ZKZ /KusJDXgVbJl8HSUg3Q/U45KyzXNVDntRtCPoB34yPbvV4MWBWIXtBe8mQyMNtOHvRVcDfRkQHQ=

[chukka]

FYI- To workaround this issue, we moved all used (by our team) bitnami charts under https://charts.jfrog.io (powered by artifactory)

if you are using artifactory , you can try Remote Repositories to cache all bitnami charts

[alemorcuq]

I can't reproduce the issue on my side:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

[alemorcuq]

I reproduced the issue now, but it worked immediately after:

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "bitnami" chart repository (https://charts.bitnami.com/bitnami):
    stream error: stream ID 1; INTERNAL_ERROR; received from peer
Update Complete. ⎈Happy Helming!⎈

$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈

Our team is looking into this.

[ArgonV]

Howdy all, I would just like to report that I am seeing this issue to, in southeast Texas.

Clone failed: Cloning into 'management-state/catalog-cache/a68a354dfd35755d14bebd02d825feb24ecbbd0496438ceb36691e1eabaa4947'... fatal: Remote branch master not found in upstream origin : exit status 128

Adding via command line still works - but in the above, this is through the Rancher Global Catalog UI.

[carrodher]

Please note this is a totally different and unrelated issue. Helm client doesn't work with git not cloning any Git repository. It seems something totally different and unrelated to Bitnami.

Regarding the original issue related to the Helm registry, we are doing more research on our side in order to find the root cause of those transient errors and mitigate them.

[ArgonV]

Ah, well the original message I had was: Update failed: fatal: not a git repository (or any parent up to mount point /var/lib) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). : exit status 128

When I totally removed the repo and added it back, the error I got was what I posted here in my previous comment. Nothing in my environments has changed (I see the same issue in Test/Dev/Pre/Prod K8s cluster envs) - that just popped up this week. The setup had been running fine for a couple of years now.

[jbianquetti-nami]

Please, note that Helm charts it's not a Git repository but a CDN distribution. It doesn't understand any git API call, and this was causing a lot of errors from the CDN perspective. We set a few rules to prevent these calls from polluting the CDN statistics and thanks to this we're able to find the root issue, that was related to caching settings. We're actively monitoring the issue.

[ArgonV]

I am not sure what Rancher UI is doing then, I'm not specifically telling it that this is a Git repo. Rancher has these requrements for adding custom Helm repos:

Custom Git Repository:

The Git URL needs to be one that git clone [can handle](https://git-scm.com/docs/git-clone#_git_urls_a_id_urls_a) and must end in .git. The branch name must be a branch that is in your catalog URL. If no branch name is provided, it will default to use the master branch. Whenever you add a catalog to Rancher, it will be available almost immediately.

Custom Helm Chart Repository:

A Helm chart repository is an HTTP server that contains one or more packaged charts. Any HTTP server that can serve YAML files and tar files and can answer GET requests can be used as a repository server.

Helm comes with a built-in package server for developer testing (helm serve). The Helm team has tested other servers, including Google Cloud Storage with website mode enabled, S3 with website mode enabled or hosting custom chart repository server using open-source projects like [ChartMuseum](https://github.com/helm/chartmuseum).

In Rancher, you can add the custom Helm chart repository with only a catalog name and the URL address of the chart repository.

So perhaps the new filtering that is being done now is preventing the clone? Or preventing Rancher from doing some check against the Helm repo, so it cannot continue? I will see if Rancher supports CDN distros specifically - I suspect they do already however.

[dboc]

@ArgonV Hi, I have the same issue here, i've been using bitnami charts in Rancher and everything was fine until last week.

The message error is the same: Clone failed: Cloning into 'management-state/catalog-cache/a68a354dfd35755d14bebd02d825feb24ecbbd0496438ceb36691e1eabaa4947'... fatal: Remote branch master not found in upstream origin : exit status 128

Did you have some new about it?

[ArgonV]

Hello @dboc , sadly I do not. It is still broken for us (after adding, deleting, refreshing, removing the branch name from the Catalog add UI, etc...) - and I'm pretty sure the new filtering in-place on the CDN distro is the cause. Other Helm 3 CDN repos add just fine in the Rancher UI.

[dboc]

@ArgonV , thanks. There is an open Issue in Ranchers Github's, Maybe its related to the Rancher and not related to bitnami. Link: https://github.com/rancher/rancher/issues/37136

[carrodher]

Apart from the Rancher UI issue, are you still facing this issue once applied on our side the mitigation described by @jbianquetti-nami at https://github.com/bitnami/charts/issues/8433#issuecomment-1086032159?

Please, note that Helm charts it's not a Git repository but a CDN distribution. It doesn't understand any git API call, and this was causing a lot of errors from the CDN perspective. We set a few rules to prevent these calls from polluting the CDN statistics and thanks to this we're able to find the root issue, that was related to caching settings. We're actively monitoring the issue.

[ArgonV]

@carrodher I changed the URL to https://raw.githubusercontent.com/bitnami/charts/index/bitnami as was suggested in https://github.com/rancher/rancher/issues/37136#issuecomment-1086704276 , refreshed and that worked for me for now.

[carrodher]

Thanks for letting us know, let's see if this workaround fixes the issue in Rancher. We are monitoring any issue using the Helm client

[rjbaat]

@carrodher I run into the same problem, I added the url https://github.com/bitnami/charts to get it to refresh in rancher UI. But then I run into the dependency problem of the mariadb chart I wanted to upgrade. That's pointing to the url: https://charts.bitnami.com/bitnami

dependencies:
- name: common
  repository: https://charts.bitnami.com/bitnami
  version: 1.13.0

[carrodher]

About the Rancher issue it was already reported in the Rancher repo, not sure how we can help with this specific topic.

[gmanera]

@carrodher I changed the URL to https://raw.githubusercontent.com/bitnami/charts/index/bitnami as was suggested in rancher/rancher#37136 (comment) , refreshed and that worked for me for now.

I've the same issue. The https://raw.githubusercontent.com/bitnami/charts/index/bitnami worked for me!! Thanks

[unnamedsos]

Hi, In our case the error is:

Update failed: fatal: not a git repository (or any parent up to mount point /var/lib) Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set). : exit status 128

this one however worked for us: https://raw.githubusercontent.com/bitnami/charts/index/bitnami

[carrodher]

This seems a specific error related to rancher, git is not involved in any action related to the Helm registry when using the helm CLI

[unnamedsos]

This seems a specific error related to rancher, git is not involved in any action related to the Helm registry when using the helm CLI

Yeah, thought the same, but it only fails at Bitnami repository! Also, the other link also works, which makes it really suspicious....

[droopy4096]

Looking at comments here https://github.com/bitnami/charts/issues/8433#issuecomment-1010946079 it looks like 9.1% failure rate is observed and indeed that is similar to what we're seeing with our CI pipelines. Aside from caching charts locally or several retries - any other suggestions?

Along the same lines: is it possible that outages happen when charts are being published and indexes being re-generated?

[arendjantetteroo]

Since this afternoon this is a continous issue for us on a gitlab ci/cd env. Locally this worked fine though. Is there any quota system in place with regards to shared ip ranges for the ci providers that now suddenly kicks in?

[pdf]

It looks like some CloudFront frontends are running terribly slowly - for my locally resolved CF frontends, 2 out of 4 run at ~50KiB/s, and the other 2 run at multi-megabytes per sec.

So 50% of the time based on DNS round-robin, index downloads will take ~5 minutes, and hence time out for any automated process (or interactive process with a sane timeout, like helm repo update).

[carrodher]

Thanks for letting us know, we are looking into it. In fact, we are also facing this same issue in our internal test & release pipelines

[jhaumont]

Hello I confirm the issue on my side. Since Thursday 26th, we have multiple alerts on helmCharts reconciliation with Flux v2. The common point of these charts? They use dependencies charts from https://charts.bitnami.com/bitnami. The error we have on kubernetes (k describe helmcharts -n <namespace> <chart name>):

dependency build error: failed to add remote dependency 'common': failed to load index for 'common': failed to strategically load index: failed to cache index to temporary file: stream error: stream ID 1; INTERNAL_ERROR; received from peer
dependency build error: failed to add remote dependency 'common': failed to load index for 'common': failed to strategically load index: failed to cache index to temporary file: context deadline exceeded (Client.Timeout or context cancellation while reading body)

[KqLLL]

I'm having the same issue.

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peer

[Nexus2k]

We've also started seeing these errors since a few days ago. Please look into it! Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: stream error: stream ID 1; INTERNAL_ERROR; received from peer

Fortunately retrying usually fixes the problem.

[wiardvanrij]

We are experiencing the same issue. Any help we can offer?

[michaelmohamed]

I just got this error minutes ago.

[mkmad]

we are also facing the same issue

[parkedwards]

we are as well. our cicd can't get through a workflow because we pull down 2-3 bitnami charts + we're hitting this error

[glloydcorecard]

Also facing this issue