bitnami / charts

Bitnami Helm Charts
https://bitnami.com
Other
9k stars 9.21k forks source link

Request/issue: release cadence #30136

Open bh-tt opened 1 day ago

bh-tt commented 1 day ago

Name and Version

all

What is the problem this feature will solve?

This really belongs in a discussion, but since this project does not have them enabled I'll open a feature request.

I'm from a small IT team managing a medium-sized infrastructure (several clusters, from 6 to 15 nodes large nodes in size). We started migrating to Kubernetes a while back, and bitnami's chart really helped us setup a lot of basic infrastructure, such as metrics-server, an etcd, postgres databases and metric-based alerting. The charts are comprehensive and well-documented, and we added a few features ourselves as well. However, after a few years we are moving away from bitnami charts for most of our deployments, replacing most releases with kustomize bases directly from the deployed application or even writing our own plain YAML files. Why?

Because of the release cadence. For production-critical infrastructure like a postgres database or argo-workflows controller, updating (and suffering downtime and/or update problems) very often is hard. Bitnami releases very often, even for projects that themselves are very stable (e.g. metrics-server, etcd, postgres, with a few to 10 or so releases per year). We found ourselves spending 1-2 hours a week updating bitnami charts, most of which had no application updates and so did not really need an update. To give a few examples:

Why does bitnami release this often? I think there are a few reasons:

I made this issue mostly to create some awareness about the impact of a high release cadence on users, especially for charts that are hard to upgrade without supervision.

What is the feature you are proposing to solve the problem?

I'm not sure there is an easy solution here. Using a helm chart means you'll always have more updates than by simply writing your own YAML files (assuming the helm chart also releases with each application update), but I'd expect a multiplier of 2-3, not 5-15.

There are a few things that can be done however:

What alternatives have you considered?

I'd like to stress this is not a criticism of the charts themselves. Consider this more an alert for something that may cost this repository users in the long term. The charts are very good, well-maintained and have a template variable for everything you can think of. But, when releasing charts for applications that are often critical, it may be good to consider a slower release cadence than 'whenever there is any change'.

javsalgar commented 1 day ago

Hi! First of all thank so much for this detailed feedback! Also thank you for using Bitnami for all these years and having contributed with features to some of our charts. We're definitely sorry to see you go. Let us share our views on the current release process:

As soon as I saw your post, I immediately thought: "what's preventing them to choose their own release cadence?" That is, for example discard all patch updates and go only for minors, or maybe go for updates that change the appVersion, and so on. If possible, could you share with us the process you followed when a new chart version was released? Something tells me that a major pain point was the lack of proper visibility on the changes (especially those automated). In order to mitigate this, we began using git tags and changelogs so you could easily see the changes between two chart versions. For example:

git log  wordpress/19.4.1..wordpress/22.2.1 --oneline bitnami/wordpress

You can also use the changelog with automation tools like renovate or dependabot, making the process easier. However, we believe that we could do better with the changelog entries (especially with the automated ones). I was thinking that, if we had a consistent, standard way of titling the changes, for example: "[bitnami/metrics-server] chore(security): Fix CVE-XXXX-YYYY", "[bitnami/metrics-server] feat: Add PDB support", you could easily track and discard releases, therefore creating your custom update cadence tailored to your needs.

We would love to hear your thoughts on this. Maintaining your own YAML can definitely be painful too, so it would be great to know if changes like the above (improve the visibility of changes and review Go applications) could help you reconsider your decision.

Once again, thank you for your feedback and continued usage through all these years.

bh-tt commented 1 day ago

We have 2 processes for updates:

We started with the 2nd setup and only later discovered a better GitOps workflow.

Better release notes would certainly help us here to determine whether an update is worth doing for our ansible playbooks. That still means reviewing quite a few release notes on a weekly basis or so to see if there are security-related updates, but it would make a difference. Right now our process consists mostly of 'open the history of the relevant Chart.yaml file and see if there are relevant changes', which is a lot of manual browsing. This is necessary as security updates for the container/app itself are often patch updates.

I'm also glad to see there is ongoing work to update Go apps to scratch containers.

We are trying to keep our own YAML to a minimum, with a few rules:

This leaves simpler apps like oauth2-proxy and the like to be written in our own YAMLs. I just tried migrating that one, and we ended up with fewer lines of YAML with our own kustomize base and a few overlays compared to actually templating the yaml.