Bitnami CDN served index.yaml planned redirection

carrodher commented 1 year ago

As pointed out in the Bitnami Helm Charts Available as OCI Packages for Early Adopters from January and later in Bitnami Helm Charts Now Generally Available as OCI Packages from April blog posts, we are involved in a gradual process for deprecating Helm chart consumption via index.yaml in favor of Helm charts as OCI packages in DockerHub. (+info about Helm OCI support)

The plan is to redirect all the traffic from the index.yaml served from the Bitnami Content Delivery Network (CDN) to a new index.yaml that is generated and stored in a reliable CDN.

⚠️ We will be moving index.yaml to a reliable CDN but we would like you to note that any possible rate limits that the CDN provider might choose to apply to individual IP addresses consuming Helm Charts at a large scale are out of our control. In addition to this, keeping all the limitations and issues faced when using large Helm chart catalogs in a single index, we strongly recommend users start consuming Helm Charts as OCI packages going forward.

🏳️ Please note the redirection will be internally implemented, end users don't need to change the repository from where charts are fetched.

In the same way, note the whole index.yaml is being generated and stored in GitHub, not in the CDN. You can find it in the archive-full-index branch. If you want to keep using older versions and the size or speed is not critical, feel free to use this one.

$ helm repo add bitnami-full-index https://raw.githubusercontent.com/bitnami/charts/archive-full-index/bitnami
"bitnami-full-index" has been added to your repositories

Following the roadmap communicated in the above-mentioned blog posts, there will be some temporary and gradual redirections:

During this period, we will perform some planned redirections from the index.yaml stored at https://charts.bitnami.com to the index.yaml stored in a different CDN. The purpose of this action is to redirect different amounts of traffic to detect any kind of issue. For end users, this means the Helm CLI commands or any CI/CD tool will fetch the index.yaml from a different location instead of from the traditional Bitnami repository.

Provisional dates and test parameters:

Traffic redirection 1: 100% of traffic during 4 hours at the end of May

Traffic redirection 2: 100% of traffic during 8 hours in mid-June

Traffic redirection 3: 100% of traffic during 24 hours at the end of June

Timeline

The first one took place on 31st May, from 10:00 CET to 14:00 CET (hours are approximate and may vary slightly) without any major issue detected ✅
The second one took place on 20th June, from 10:00 CET to 18:00 CET (hours are approximate and may vary slightly) without any major issue detected, just an empty index.yaml was returned during the first minutes ✅
The third redirection took place on 29th June from 14:00 CET to 30th June at 14:00 CET, without any major issue detected, due to that, the redirection wasn't undone and the new CDN will continue to serve the index.yaml ✅

carrodher commented 1 year ago

This planned redirection was performed without issues, the next one is planned for mid-June and will be active for 8 hours. We'll edit this pinned issue with more exact information once the date is near.

carrodher commented 1 year ago

The second redirection will take place tomorrow (20th June) from 10:00 CET to 18:00 CET

vilmosnagy commented 1 year ago

Hi @carrodher, we experience some issues:

✦ ❯ helm repo update bitnami
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "bitnami" chart repository (https://charts.bitnami.com/bitnami):
    empty index.yaml file

is this related to this redirection?

christiangonre commented 1 year ago

Hi @carrodher, we experience some issues:

✦ ❯ helm repo update bitnami
Hang tight while we grab the latest from your chart repositories...
...Unable to get an update from the "bitnami" chart repository (https://charts.bitnami.com/bitnami):
  empty index.yaml file

is this related to this redirection?

I was facing this issue and it after a few minutes it looks like it has been solved.

carrodher commented 1 year ago

It was related yes, now it should be solved. https://charts.bitnami.com/bitnami/index.yaml should redirect to https://repo.vmware.com/bitnami-files/index.yaml without any issue right now

carrodher commented 1 year ago

The second planned redirection was performed without major issues, just an empty index.yaml was returned during the first minutes. The redirection is now disabled until the next action

Techassi commented 1 year ago

I don't know if this is related, but I encounter this error when adding the repo (https://charts.bitnami.com/bitnami/) via Helm more than I would like: error converting JSON to YAML: yaml: control characters are not allowed

carrodher commented 1 year ago

I don't know if this is related, but I encounter this error when adding the repo (https://charts.bitnami.com/bitnami/) via Helm more than I would like: error converting JSON to YAML: yaml: control characters are not allowed

It shouldn't be related but in order to reproduce the issue on our side, could you please provide more info about the exact command you're running when that error appears? What is the cadence of the issue?

Techassi commented 1 year ago

Okay. That's good to hear! I actually don't use the Helm CLI tool directly, but use a Go library to programmatically add chart repos. The Helm client in use is github.com/mittwald/go-helm-client. The gist of it looks something like this:

import (
  gohelm "github.com/mittwald/go-helm-client"
  "helm.sh/helm/v3/pkg/action"
  "helm.sh/helm/v3/pkg/repo"
)

options := gohelm.Options{
  Namespace: "default",
  Debug:     false,
}

helmClient, err := gohelm.New(&options)
if err != nil {
    panic(err)
}

chartRepo := repo.Entry{
  Name: name, // bitnami
  URL:  url,  // https://charts.bitnami.com/bitnami/
}

if err := helmClient.AddOrUpdateChartRepo(chartRepo); err != nil {
  panic(err)
}

I would guess that roughly 40% of the time I try to add the repo the above mentioned error occurs. The behaviour is not predictable. On one run the addition fails, on the next run (a few seconds later), it works without any issues.

Lmk if I should open a separate issue for this!

carrodher commented 1 year ago

Redirections to a new CDN were enabled and disabled during some specific time slots, if you're still facing issues outside those time slots, definitely, it is not related to the redirection. In fact, we are looking for a different CDN since we are facing some issues with the current one, hopefully, the new one is more stable in that sense.

carrodher commented 1 year ago

The third redirection will take place today (29th June) from 14:00 CET to tomorrow (30th June) at 14:00 CET, although if no major issues are discovered, the redirection won't be undone and the new CDN will continue to serve the index.yaml from then on.

carrodher commented 1 year ago

The third redirection took place on 29th June from 14:00 CET to 30th June at 14:00 CET, without any major issue detected, due to that, the redirection wasn't undone and the new CDN will continue to serve the index.yaml ✅

bpsullivan3 commented 1 year ago

Hi there, I am surprised no one has mentioned the same issue, but it appears this change broke our Helm packaging process.

We add the repo using a corporate CA file, and an error is thrown each time we attempt to connect to the new CDN due to a failed TLS hostname verification

$ helm repo add bitnami https://charts.bitnami.com/bitnami --ca-file ca.pem

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://repo.vmware.com/bitnami-files/index.yaml": x509: certificate is valid for *.vmware.com, vmware.com, not charts.bitnami.com

I attempted to get around this, at least temporarily, by adding the --insecure-skip-tls-verify flag (which is not an ideal solution), but that didn't seem to help either...

$ helm repo add bitnami https://charts.bitnami.com/bitnami --ca-file ca.pem --insecure-skip-tls-verify

Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: failed to fetch https://charts.bitnami.com/bitnami/index.yaml : 403 Forbidden

Clearly charts.bitnami.com was never added as a SAN on the new CDN's certificate (which I can confirm pretty quickly is the case), so it makes sense to me that this error would occur.

Any advice on how to proceed here?

carrodher commented 1 year ago

Hi @bpsullivan3, thanks for reporting this issue. The same issue was already reported by another user at https://github.com/bitnami/charts/issues/17408#issuecomment-1614510914, but it seems it was solved on his end.

What is the Helm version you're using and the location from where the repo is being fetched?

bpsullivan3 commented 1 year ago

We are using Helm v3.8.0, and the repo is being fetched from a Gitlab Runner VM based in the US-West region.

carrodher commented 1 year ago

We are not able to reproduce the issue in the different tests we did and are doing right now. As a proposal, could you try using the latest Helm CLI version (3.12.1)? Maybe the issue is something related to the Helm client and solved in recent versions.

bpsullivan3 commented 1 year ago

Unfortunately the latest Helm version still does not work for me even testing locally

$ helm version 
version.BuildInfo{Version:"v3.12.1", GitCommit:"f32a527a060157990e2aa86bf45010dfb3cc8b8d", GitTreeState:"clean", GoVersion:"go1.20.4"}

$ helm repo add bitnami https://charts.bitnami.com/bitnami --ca-file ca.pem
Error: looks like "https://charts.bitnami.com/bitnami" is not a valid chart repository or cannot be reached: Get "https://repo.vmware.com/bitnami-files/index.yaml": tls: failed to verify certificate: x509: certificate is valid for *.vmware.com, vmware.com, not charts.bitnami.com

I actually can find an open PR on Helm regarding what sounds like a very similar issue

carrodher commented 1 year ago

It seems related to the issue associated with the PR you mentioned. Currently, the default method to fetch Bitnami charts is using the Bitnami DockerHub OCI repository, and given that OCI is supported out of the box in Helm 3.8.0, could you try using OCI? See https://blog.bitnami.com/2023/04/httpsblog.bitnami.com202304bitnami-helm-charts-now-oci.html for more information about this topic

bpsullivan3 commented 1 year ago

It seems that is the next logical move. We were hoping to avoid switching to OCI just yet, as it requires a non-trivial amount of changes to our build process, however that might be unavoidable at this point.

bitnami / charts

Bitnami CDN served index.yaml planned redirection #16945

Timeline