search

bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
8.96k stars 9.19k forks source link

[bitnami/kube-prometheus] UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v1" is invalid: data: Too long: must have at most 1048576 bytes #29876

Open felipewnp opened 1 week ago

felipewnp commented 1 week ago

Name and Version

bitnami/kube-prometheus:9.6.3

What architecture are you using?

amd64

What steps will reproduce the bug?

Create a few prometheus scrapping config and prometheus rules and add them to values.yaml: Launch helm with those configs

Are you using any custom parameters or values?

A LOT, I don't think it's relevant though.

What is the expected behavior?

Installation successfull

successful

What do you see instead?

Release "kube-prometheus" does not exist. Installing it now. Error: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v1" is invalid: data: Too long: must have at most 1048576 bytes

Additional information

Yes, this is pretty much a copy of #28990, wich was closed without a solution.

I'm bringing this over again because I too am suffering from this problem.

I've reached a point where I cannot install/upgrade to versions past 9.0.9 of the chart and I'm stuck in an outdated chart/Prometheus version.

I'm thinking of downloading the chart locally and pretty much include "everything" in a .helmignore file since I don't use the rollback feature of helm.

The thing is, I don't know if this works.

Can the chart be upgraded later?

There's an option to store the helm chart config in PostgreSQL, but is a beta feature and I only have a problem with this specific chart. So, not really an option.

The more realistic option (in my case at least), would be to stop using the project and go for a separate alertmanager, blackbox, node-exporter, etc, if you reach this point.

But it's very painful to have to start with kube-prometheus and have to drop everything at some point.

So, if nothing works, this issue should serve at least as an alert:

If you plan to customize too much this chart, don't.

Since Kubernetes has a limit of 1MB for a secret and is mandatory that scrapping configs and rules are stored in a Secret is almost impossible to have a production configuration.

The problem is not a high volume of rules or scrapping configs. The problem I think is the chart size itself.

Helm sends the whole chart as a secret to maintain history and it's very easy to reach this point.

javsalgar commented 1 week ago

Hi,

Sorry for the situation. If you move the PrometheusRule objects outside of the chart, are you able to bypass the issue? It's not ideal, but at least it could help unblocking the upgrade.

Leaving the upstream helm ticket for reference https://github.com/helm/helm/issues/9788

felipewnp commented 6 days ago

I'm not deploying rules inside the chart, only the applications.

Deploying those with a custom alertmanager template is enough to exceed the 1MB secret limit.

I'll do more tests today to see if I can do something about it because the thought of re-doing everything in separate helm charts is unbearable...

felipewnp commented 6 days ago

@javsalgar I've done some digging...

It looks like these files aren't included in the .helmignore file:

The crds/ folder alone weighs 3.7MiB.

I did not find any mention of the bash script hack/update_crds.sh or the crds/ folder in the helm chart.

Can we add them to the .helmignore file?

Removing them, reduced the helm secret to 70852 bytes.

javsalgar commented 6 days ago

Hi!

I believe the update_crds.sh script could be included in the helmignore for sure.

About the crds, this is deployed automatically by helm, so I don't think we can add it to the helmignore as it would stop working on a fresh installation.

I will check about the README.md file.

What happens if you don't include these two? Is there any significant change in the secret? I suspect that the changes you obtained are because of the crds folder I'm afraid

felipewnp commented 5 days ago

What happens if you don't include these two? Is there any significant change in the secret? I suspect that the changes you obtained are because of the crds folder I'm afraid

I didn't test without the CRD's folder indeed.

Now I see that the crd/ folder is a default folder for Helm. That explains why I didn't find any mentions of it inside the chart.

I think the more optimal solution would be creating a separate helm chart for the CRD, like Karpenter does.

I will try a new deploy removing README.md and hack/update_crds.sh, but they are very small to make a difference..

felipewnp commented 5 days ago

@javsalgar, with README.md and hack/ inside the .helmignore, the chart release secret is about 988848 bytes.

javsalgar commented 4 days ago

Checking with the team, removing the README.md from the tarball would break some automation and integrations with other systems so right now I'm afraid this is not possible. We will continue checking what to do, but at the moment please use that workaround for your use case. Sorry for the inconvenience.

felipewnp commented 3 days ago

@javsalgar Can you provide a little more info on what the README.md is used for so I can check if it is relevant to my case?

Also, can't we separate the crd's in another helm chart?

It wouldn't be a breaking change since it wouldn't affect current installations.

. . .

For anyone out there, I think the easiest solution is to download the chart, include the crd's folder in the .helmignore file, and apply the crd's before installing the chart.

This way you'll have lots of space in the helm secret.

javsalgar commented 3 hours ago

Hi,

I'd say that the README.md is not relevant to your case, as it is required with integrations for other distribution platforms and generation of some internal documentation.

The use of CRDs in a separate chart would be an exception in our automations and I'm not sure if it is viable at the moment. We'll need to evaluate it.