[bitnami/kube-prometheus] UPGRADE FAILED: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v3" is invalid: data: Too long: must have at most 1048576 bytes

[rmacian]

Name and Version

bitnami/kube-prometheus:9.5.12

What architecture are you using?

amd64

What steps will reproduce the bug?

1. Create a few prometheus scrapping config and prometheus rules and add them to values.yaml:
2. Launch helm with those configs

Are you using any custom parameters or values?

yes, feeding the prometheus rules with a jinja template

    additionalScrapeConfigs:
      enabled: true
      type: internal
      internal:
        jobList: "{{ prometheus_scrape_config }}"
    additionalPrometheusRules: "{{ prometheus_rules }}"

What is the expected behavior?

Installation successfull

What do you see instead?

Release "kube-prometheus" does not exist. Installing it now. Error: create: failed to create: Secret "sh.helm.release.v1.kube-prometheus.v1" is invalid: data: Too long: must have at most 1048576 bytes

Additional information

Since kubernetes has a limit of 1Mb for a secret and is mandatory that scrapping configs and rules are stored in a Secret is almost impossible to have a production configuration. The problem is not a high volume of rules or scrapping configs. I can render my values.yaml with helm template and it's only about 112kb and I can apply it with kubectl. The problem I think is the chart size itself. Helm sends the whole chart as a secret to mantain history and it's very easy to reach this point.

Why cannot be stored in a configmap to avoid this limitation?

[javsalgar]

Hi,

Thank you for reporting! This is something that has appeared in our testing too and we're trying to find a solution. About storing it in a configmap, that's something to be answered by the helm developers, as this is as limitation on their side. I imagine that, by default, they may to store it in a secret because there may be passwords implied.

[github-actions[bot]]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.