search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
9.08k stars 9.25k forks source link

Helm charts repository index.yaml retention policy #10539

Closed carrodher closed 1 year ago

carrodher commented 2 years ago

As reported in this issue (https://github.com/bitnami/charts/issues/8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml. This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

One of the alternatives considered was the use of compression at CloudFront, in that case, this solution doesn’t work since compression is not used by the Helm client (helm) itself (see https://github.com/helm/helm/pull/8070) so it doesn’t solve the reported issue.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

⚠️ Please note this action is not removing/deleting any Helm chart, packaged tarballs (.tgz) won't be removed, this action is only affecting index.yaml used to list the Helm charts. Previous versions of the index.yaml can be used to install old Helm charts.

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (https://github.com/bitnami/charts/pull/10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

🔧 Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml.

DmytroDrachov commented 2 years ago

Thank you very much

sisrael-dn commented 2 years ago

Hi, https://charts.bitnami.com/bitnami seems to return 403 AccessDenied error at the moment. Is this related to this issue? My guess is that if you work with S3 bucket as CloudFront origin, the new index.yaml got the wrong ACL now (not publicly readable)

potiuk commented 2 years ago

This is TERRIBLY wrong. For example, we released Helm Chart to our users https://airflow.apache.org/docs/helm-chart/stable/index.html with bitnami's Postgres 10.5.3 and all the charts our users are using are useless now if they use embedded Postgres.

How is it possible you do such a breaking change in such a way ? This must be some kind of a joke.

This totally undermines any trust in bitnami as a source of charts - You basically rendered all historical charts released by likely huge number of projects useless.

bschurig commented 2 years ago

As reported in this issue (#8433), in the last few times we are facing some issues with the index.yaml associated with the Bitnami Helm charts repository.

Current situation

After some investigation, it seems the root cause is related to CloudFront reaching some limits due to the volume of traffic when serving the index.yaml. This index.yaml contains all the Bitnami Helm charts history (around 15300 entries), producing a pretty fat 14MB file. Given the size of the file and the volume of traffic, thousands of terabytes of download traffic per month are being generated.

Mitigation

As the first line of action, we will reduce the size of the index.yaml by removing some old versions and keeping all versions for a period of time (6 months).

Please note Helm charts tarballs (.tgz) won't be removed, this action is only affecting index.yaml.

Result

Applying this approach (#10530), we obtained the following results:

Total chart versions
* Before: 15260
* Removed: 12138
* After: 3122

Producing a reduced 3.5MB index.yaml.

Workaround for previous versions

The index.yaml is stored in this repository under the index branch, users should be able to use any commit in that branch to add a previous version of the index.yaml, for example:

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

It would be fantastic if someone could get this added to the readme.

GMZwinge commented 2 years ago

Instead of removing all versions older than 6 months, would it be possible to keep the last major/minor versions older than 6 months, eg: mongodb-9.2.6.

beltran-rubo commented 2 years ago

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

adawalli commented 2 years ago

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

Why didn't you consider a proper CDN or basically...anything else? You make it seem like this is no big deal and that there is an existing easy path for people. The reality is that a ton of pipelines, dev flows, and deployments have just broken and people are scrambling to find out why. There is no way to spin that this was a good decision as executed - this was done extremely rashly.

CrisFavero commented 2 years ago

The Url in this code snippet for your fix returns 404 

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories
nabilbendafi commented 2 years ago

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami "bitnami-pre-2022" has been added to your repositories

@CrisFavero Normal. Because helm will look for an index.yaml in this base url

adawalli commented 2 years ago

The Url in this code snippet for your fix returns 404

$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

redacting incorrect info - the above info is correct

nabilbendafi commented 2 years ago

@CrisFavero

A chart repository is an HTTP server that houses an index.yaml file and optionally some packaged charts. When you're ready to share your charts, the preferred way to do so is by uploading them to a chart repository.

So the current workaround simply uses githubusercontent.com as a free webserver to provide that essential/critical index.yaml, that contains the links to the helm charts versions we all need.

Pokom commented 2 years ago

@adawalli if you don't specify the commit hash then you're using the latest index, which does not include older releases. @CrisFavero this URI worked for me: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

adawalli commented 2 years ago

@adawalli if you don't specify the commit hash then you're using the latest index, which does not include older releases. @CrisFavero this URI worked for me: https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Thanks - you are correct - I redacted my previous comment to prevent confusion

carrodher commented 2 years ago

In order to list old versions you need to use a previous commit, see below the differences:

$ helm repo list
NAME                URL
bitnami             https://charts.bitnami.com/bitnami
bitnami-index       https://raw.githubusercontent.com/bitnami/charts/index/bitnami
bitnami-pre-2022    https://raw.githubusercontent.com/bitnami/charts/pre-2022/bitnami
spmse commented 2 years ago

@adawalli the information you provided seems to be correct for both cases:

When first trying the workaround to add the pre-2022 repo I got the following results:

$ helm repo add bitnami-pre-2022 \
  https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Error: looks like 
"https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami" 
is not a valid chart repository or cannot be reached: 
error unmarshaling JSON:
  while decoding JSON: 
    invalid character 'd' looking for beginning of object key string

When retrying the exact same command at a later point in time I got this result:

"bitnami-pre-2022" has been added to your repositories

Right now we can access the required charts again, but since there was different behavior this seems a bit unstable to me 🤔 .

jfklingler commented 2 years ago 
$ helm repo add bitnami-pre-2022 https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami
"bitnami-pre-2022" has been added to your repositories

Isn't this just moving the problem from CloudFront to Github? It's a large file no matter who's serving it.

obrunodelgado commented 2 years ago

Oh my god, seriously that you did it?

dunn commented 2 years ago

Was there truly no way to negotiate with AWS on the bandwidth limits?

At the very least, as @GMZwinge said, you should be providing the latest of each historical major release.

nodesocket commented 2 years ago

What's going on? Just today started getting:

Error: can't get a valid version for repositories postgresql. Try changing the version constraint in Chart.yaml

Chart.yaml is:

  - name: postgresql
    version: 10.3.18
    repository: "@bitnami"

This is absurd, can't just break everything for people. Additionally I'm not going to add a helm repo pointing to a GitHub raw commit.

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Bitnami this is horrible developer experience.

potiuk commented 2 years ago

Hi @potiuk, we apologise for the inconvenience. The Helm charts you built & distributed previously still are working properly for users, the previous version of the Helm chart is included into the tarball itself.

The part that is affected is the helm package or helm dependency update actions as part of the packaging process. All the charts are publicly available, all versions. The workaround mentioned into the post allows you to continue building the Helm chart with the exact same version.

Yep. Thanks for the information - this is indeed the case., which I've learned since writing it. So the problem is indeed limited to people who are developing the charts, CI, development pipelines and generally anyone adventorous etc. Which is stil quite a big deal to get overnight change like that. This seriously undermines usefulness of using bitnami charts in the future for any serious project. But maybe that's what you wanted to achieve - this is a free offering which you can't monetize. Travis CI did similar things in the past by deprioritising free OSS support and maybe won it in the "commercial" front but lost big time (mainly to GitHub Actions) on the OSS front. That's really business not technical decision that you probably considered while making the change.

For us the decision is made - we will simply vendor your chart in. I actually even started to discuss it day before yesterday as we saw the flakiness - but now we accelerate this move.

Also one more comment and constructive suggestion. Surprises like that might easily break reputation. Maybe reputation in the free/OSS is not the most important for VMware-owned company, but I think if you want to really solve the problem, and still keep the free offreing "serious" for the free/OSS development users, you should simply deliver each chart separately as separate repository (or basically a sub-folder of the main folder that has separate index).

This is the only sustainable way to make it works long term and much more in-line with Helm 3 approach where they recognized the problem of centralized repos. You should follow their blueprint.

The cncf foundation actually made the decision about getting rid of the centralised repo and even made a deliberate effort to reach out to the community (we got plenty of discussions with cncf folks about it) to prepare everyone for the change well in advance. This change went really went and smooth and I think that was a model approach to change management. You probably know about it but for the passers-by this is just one of the blog posts where they warned about the change. https://www.cncf.io/blog/2020/10/07/important-reminder-for-all-helm-users-stable-incubator-repos-are-deprecated-and-all-images-are-changing-location/

nodesocket commented 2 years ago

Also, the suggested link with all versions is returning 404 as well:

https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami

Any recommendations? Need to get this resolved, breaking all of our deployments.

thomasv314 commented 2 years ago

@nodesocket The suggested link is the repo URL you add via a helm repo command or in your declarative infra.

# This will 404 if you visit it in the browser
$ helm repo add https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/

Helm will then reach out to https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml, which returns a 200

wacuuu commented 2 years ago

Now that @potiuk has opened my eyes to the monetization topic(thank you very much for that, this is something I always forget), maybe this is some crazy decision related to vmware aquisition by broadcom? This is conspiracy theory, I know, but yeah, money in the CDN mentioned in the discussion maybe?

nodesocket commented 2 years ago

@thomasv314 thanks. I tried the following:

$ helm repo remove bitnami
"bitnami" has been removed from your repositories

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/
"bitnami" has been added to your repositories

$ helm repo update

But now getting:

Error: no cached repository for helm-manager-54d2620bbb6f1bb3f35d4c7f945bfa25077949488dcbb0a4d01c90f2c35baa59 found. (try 'helm repo update'): open /home/ubuntu/.cache/helm/repository/helm-manager-54d2620bbb6f1bb3f35d4c7f945bfa25077949488dcbb0a4d01c90f2c35baa59-index.yaml: no such file or directory

Running Helm 3.8.2:

$ helm version
version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}
potiuk commented 2 years ago

Now that @potiuk has opened my eyes to the monetization topic(thank you very much for that, this is something I always forget), maybe this is some crazy decision related to vmware aquisition by broadcom? This is conspiracy theory, I know, but yeah, money in the CDN mentioned in the discussion maybe?

I wouldn't think of it as "conspiracy" theory. I can totally relate to "free offering having no guarantees" - no complaints on it. And if that's a business decision to be less "free users" friendly, so be it - this perfectly understandable, just painful for those who put their trust in the free service (wiithout any guarantees of course). As an OSS maintainer - I can totally relate to that but while we provide no guarantees, we at least document and agree to policies that we as "free software maintainers under the Apache Software Foundation umbrella" can communicate to our users well in advance (and yeah - we also relentlessly refuse any kind of support when someone "demands" free support beyond those policies).

So this is more the matter of some kind of expectations the users have - commmunicating them and sticking to it. If they are undocumented, there are some implicit assumptions. Changing (or actually introducing) policies overnight without earlier communication and warnings which are effective immediately is one of the worst ways I can imagine to get strong, supportive community and user's trust. That's it.

Reputation can be built for years and lost overnight (especially if there is a lack of self-reflection and responding to your users).

nodesocket commented 2 years ago

Now, breaking on RabbitMQ, even with using helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/.

The Chart.yaml for our RabbitMQ is:

annotations:
  category: Infrastructure
apiVersion: v2
appVersion: 3.9.8
dependencies:
  - name: common
    repository: https://charts.bitnami.com/bitnami
    tags:
      - bitnami-common
    version: 1.x.x
  - name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami
description: Open source message broker software that implements the Advanced Message Queuing Protocol (AMQP)
engine: gotpl
home: https://github.com/bitnami/charts/tree/master/bitnami/rabbitmq
icon: https://bitnami.com/assets/stacks/rabbitmq/img/rabbitmq-stack-220x234.png
keywords:
  - rabbitmq
  - message queue
  - AMQP
maintainers:
  - email: containers@bitnami.com
    name: Bitnami
name: rabbitmq
sources:
  - https://github.com/bitnami/bitnami-docker-rabbitmq
  - https://www.rabbitmq.com
version: 8.24.1

Error message is:

Error: can't get a valid version for repositories rabbitmq. Try changing the version constraint in Chart.yaml
$ helm repo list
NAME        URL
stable      https://charts.helm.sh/stable
minio       https://helm.min.io
datadog     https://helm.datadoghq.com
fusionauth  https://fusionauth.github.io/charts
bitnami     https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/
nodesocket commented 2 years ago

@carrodher any idea why RabbitMQ is still broken even when using bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/?

- name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami
cep21 commented 2 years ago

Breaking an index.yaml seems not great: it's kind of like breaking an API version without a major version change. Some alternatives

nodesocket commented 2 years ago

Keep the old index.yaml, deprecate that location (no longer give it updates), and put newer charts on a different URL with different guarantees about longevity

Yes 💯 .

aariacarterweir commented 2 years ago

This is absolutely cooked

EvanCarroll commented 2 years ago

Just out of curiosity, why not just use compression...

wget https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml

If you take that file and compress to

So basically, this doesn't seem to be a problem if compression is supported. Seems you could host it on another mirror and push out a minor version of Helm that supports compression and you'd be good. 852k is basically the size of a single-page app wth Angular or React.

I also haven't looked, but if the client supports gzip you could just rely on transparent http gziping with something like mod_deflate.

justusbunsi commented 2 years ago

Hi. Does Helm supports being redirected during index.yaml fetch? If so, would it be possible to redirect the request for index.yaml from the old location to the pre-2022 commit endpoint? That way all old versions would still be available without breaking things.

alxgomz commented 2 years ago

@carrodher Can you please share a clear plan on where you guys are going with this issue. We ned to plan accordingly in order to a) get this back up and running, b) make sure we choose to right measure for the future. Also sharing a timeline for "back-to-normal" would greatly help

Raboo commented 2 years ago

Why not use any of the hosting provides that offers free hosting for open source projects? Perhaps host the helm index on github pages?

lbornov2 commented 2 years ago

How could you possible introduce such a breaking change? This is absolutely pathetic.

soylentseth commented 2 years ago

Fuck!

kamikazechaser commented 2 years ago

You are breaking too many builds. postgresql 10.9.5 just disappeared and I had to re-write the entire values file with the new one forcing us to take care of breaking changes. It is dead simple to maintain an index for at least LTS releases for the underlying software versions.

pdf commented 2 years ago

@carrodher the right thing to do here is to immediately roll back this change, and work out how to proceed afterwards - occasional timeouts are a better situation than total breakage. Leaving the repo in the current state with users' production pipelines broken is untenable - even if there's a wonky workaround (relying on Github to act as your CDN for a pinned commit, instead of fixing the prod CDN), that's likely thousands of builds that need to be updated to continue functioning, once those affected can actually work out why their builds exploded.

thewillhuang commented 2 years ago

Rollback please, is this a joke??

rock217 commented 2 years ago

Maybe they're freeing up enough space for those ARM charts we've been waiting for.

droopy4096 commented 2 years ago

Constructive suggestion: perhaps provide OCI Helm repo? That would completely bypass index.yaml issue. Or maintain both - old-style repo trimmed down to 6 month or however long that should be to keep thing reasonable and OCI repo with everything

Alternatively perhaps going the route suggested earlier: that for anything older than 6mo keep the most recent minor release for each major one?

lbornov2 commented 2 years ago

We're still waiting for you to roll this back...

nodesocket commented 2 years ago

Just out of curiosity, why not just use compression...

wget https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml

If you take that file and compress to

  • .gz with regular gunzip you'll get a 1.1MB file
  • .xz with xz -9 gets you down to 852k

So basically, this doesn't seem to be a problem if compression is supported. Seems you could host it on another mirror and push out a minor version of Helm that supports compression and you'd be good. 852k is basically the size of a single-page app wth Angular or React.

I also haven't looked, but if the client supports gzip you could just rely on transparent http gziping with something like mod_deflate.

Indeed, compression can easily be enabled in CloudFront

Screen Shot 2022-06-03 at 9 28 46 AM

Even using the GitHub hosted commit as the Helm repo, I am still unable to helm dependency update RabbitMQ version 8.24.1 for some strange reason. Please roll this back!!!

beltran-rubo commented 2 years ago

So basically, this doesn't seem to be a problem if compression is supported

Thanks for the suggestion but, even compression is supported by CloudFront, it is not supported by Helm client right now.

nodesocket commented 2 years ago

To anybody else, the reason that postgresql worked but rabbitmq did not was in the Chart.yaml for postgresql is was using:

  - name: postgresql
    version: 10.3.18
    repository: "@bitnami"

Note, the reference to the repository as @bitnami. However rabbitmq chart was hardcoding the repository url as:

  - name: rabbitmq
    version: 8.24.1
    repository: https://charts.bitnami.com/bitnami

To fix the problem, you must also update Chart.yaml for each chart and use the dynamic repository reference of @bitnami. Thus the new rabbitmq Chart.yaml should look like:

  - name: rabbitmq
    version: 8.24.1
    repository: "@bitnami"

This still does not address the core problem of having to do:

$ helm repo remove bitnami
"bitnami" has been removed from your repositories

$ helm repo add bitnami https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/
"bitnami" has been added to your repositories

$ helm repo update
MichelZ commented 2 years ago

What about using e.g. CloudFlare? I‘m sure they can accomodate all your needs…

williambrode commented 2 years ago

What is the suggestion going forward? If we apply the workaround won't we be in the same situation again in 6 months (some of our in-use charts disappear from the current index.yaml)? I'm looking for some way to be able to update my helm charts versions in the future without changing repositories. I could set up a local service of some kind if that would work.

nodesocket commented 2 years ago

Can anybody document the process of setting up our own bitnami chart mirror? Is it a simple as:

  1. serve https://raw.githubusercontent.com/bitnami/charts/eb5f9a9513d987b519f0ecd732e7031241c50328/bitnami/index.yaml
  2. update helm repo for bitnami:
$ helm repo remove bitnami

$ helm repo add bitnami https://our-bitnami-chart-mirror.io

$ helm repo update

Or do we also need all the contents of all charts not just the index.yaml?

jfklingler commented 2 years ago

@nodesocket I think that will only serve the fixed-point-in-time index. If you want to serve all the things locally, it would be something more like:

  1. git pull upstream master
  2. helm repo index
  3. helm repo serve

That should get your local repo running. Then switching over to use that is as you described in your step 2. I haven't actually set this up yet, but I'm a-fixin' to this weekend if this isn't all reverted back to a functioning state.

That should all work fine until someone decides to delete git history because it takes a long time to clone...

nodesocket commented 2 years ago

@jfklingler thanks so much for the info. Can you report back after your work this weekend? I have a feeling I'm going to need to host my own mirror for Bitnami Charts as we need legacy versions and using a GitHub commit as a CDN is not a solution.