Request/issue: release cadence

[bh-tt]

Name and Version

all

What is the problem this feature will solve?

This really belongs in a discussion, but since this project does not have them enabled I'll open a feature request.

I'm from a small IT team managing a medium-sized infrastructure (several clusters, from 6 to 15 nodes large nodes in size). We started migrating to Kubernetes a while back, and bitnami's chart really helped us setup a lot of basic infrastructure, such as metrics-server, an etcd, postgres databases and metric-based alerting. The charts are comprehensive and well-documented, and we added a few features ourselves as well. However, after a few years we are moving away from bitnami charts for most of our deployments, replacing most releases with kustomize bases directly from the deployed application or even writing our own plain YAML files. Why?

Because of the release cadence. For production-critical infrastructure like a postgres database or argo-workflows controller, updating (and suffering downtime and/or update problems) very often is hard. Bitnami releases very often, even for projects that themselves are very stable (e.g. metrics-server, etcd, postgres, with a few to 10 or so releases per year). We found ourselves spending 1-2 hours a week updating bitnami charts, most of which had no application updates and so did not really need an update. To give a few examples:

• bitnami/etcd has had 150 releases since etcd version 3.5.4. Etcd is currently at 3.5.16, meaning there were over 12 times as many etcd chart updates as there are application updates. To give an estimate, 3.5.4 was release april 2022, meaning bitnami has updated the chart more than once a week, every week, since that moment.
• sealed-secrets has had 117 releases for 35 app updates (and sealed-secrets has been releasing less often as the controller matures). Half of those releases were for the last 8 releases of sealed-secrets
• metrics-server has has 90 bitnami releases since metrics-server 0.6.1, whereas there are only 7 application updates
• I could go on, but you get the point

Why does bitnami release this often? I think there are a few reasons:

• usage of a common helm chart: every time this updates, all other charts get an update
• tuning helm charts of course also releases a new version, and there is a lot of tuning going in within this repository
• bitnami uses a base debian image, even for applications that could easily use a FROM scratch base image, such as most Go applications (half the charts in this repository suffer from this problem). This naturally causes another release for a lot of charts whenever the base image updates, I see 7 updates since debian bookworms' release.
• some charts in this repo also depend on other charts, for example oauth2-proxy on redis or keycloak on postgres

I made this issue mostly to create some awareness about the impact of a high release cadence on users, especially for charts that are hard to upgrade without supervision.

What is the feature you are proposing to solve the problem?

I'm not sure there is an easy solution here. Using a helm chart means you'll always have more updates than by simply writing your own YAML files (assuming the helm chart also releases with each application update), but I'd expect a multiplier of 2-3, not 5-15.

There are a few things that can be done however:

• some updates could be batched, this is the approach taken by almost every large open-source project
• using distroless images helps, and I'm pretty sure I saw some information about that in this or the containers repo recently. Cannot find it right now however
• chart dependencies cannot really be helped, though those also cause a cascade whenever a base chart updates

What alternatives have you considered?

I'd like to stress this is not a criticism of the charts themselves. Consider this more an alert for something that may cost this repository users in the long term. The charts are very good, well-maintained and have a template variable for everything you can think of. But, when releasing charts for applications that are often critical, it may be good to consider a slower release cadence than 'whenever there is any change'.

[javsalgar]

Hi! First of all thank so much for this detailed feedback! Also thank you for using Bitnami for all these years and having contributed with features to some of our charts. We're definitely sorry to see you go. Let us share our views on the current release process:

• To provide some context, this release process of "one release per change" is inherited from the initial, now deprecated, "helm/charts" repository maintained by the official helm team. Bitnami was a core contributor on this repository and, when this was discontinued, we simply followed the same approach to maintain the same UX and expectations to the community.

• Having said that, we also believe that making a feature available as soon as possible is extremely helpful to the community. I can think of: Solutions to bugs, HIGH/CRITICAL CVEs fixed, useful features... In addition to this, it is helpful for us as the effort to monitor and decide when to cut releases of more than 100 charts could easily become out of hand.

• However, the cases you highlighted (etcd, metrics-server) definitely illustrate a painful point if you want to keep the charts up to date. As you mentioned, most of these Automated Releases (PRs with the title "Release ") have to do most likely with security releases or with containers with updated bash logic (the one available in the bitnami/containers repository). In the case of CVEs, we are commited to have containers (and by extension charts) with 0 fixable CVEs. Does this imply almost weekly releases of charts? We performed an analysis on the Debian release cadence in the past and we saw we would need to update every 3 days (approximately) to keep up with security issues. Now our system is more intelligent and is able to detect when there is a fixable CVE but the truth is that a close-to weekly release cadence is almost unavoidable.

• You definitely have a point on Go apps that could become scratch containers. We tend to use scratch containers whenever it makes sense and if the upstream project uses scratch too. We have on-going initiatives to review our Go apps to see if they can be changed to scratch, so you can expect some solutions to change as long as the upstream project also uses scratch/distroless.

As soon as I saw your post, I immediately thought: "what's preventing them to choose their own release cadence?" That is, for example discard all patch updates and go only for minors, or maybe go for updates that change the appVersion, and so on. If possible, could you share with us the process you followed when a new chart version was released? Something tells me that a major pain point was the lack of proper visibility on the changes (especially those automated). In order to mitigate this, we began using git tags and changelogs so you could easily see the changes between two chart versions. For example:

git log  wordpress/19.4.1..wordpress/22.2.1 --oneline bitnami/wordpress

You can also use the changelog with automation tools like renovate or dependabot, making the process easier. However, we believe that we could do better with the changelog entries (especially with the automated ones). I was thinking that, if we had a consistent, standard way of titling the changes, for example: "[bitnami/metrics-server] chore(security): Fix CVE-XXXX-YYYY", "[bitnami/metrics-server] feat: Add PDB support", you could easily track and discard releases, therefore creating your custom update cadence tailored to your needs.

We would love to hear your thoughts on this. Maintaining your own YAML can definitely be painful too, so it would be great to know if changes like the above (improve the visibility of changes and review Go applications) could help you reconsider your decision.

Once again, thank you for your feedback and continued usage through all these years.

[bh-tt]

We have 2 processes for updates:

• an automated setup with argoCD/Renovate, which automatically updates a number of less important charts and/or opens PRs when there are changes
• a number of base infrastructure charts (metrics-server, etcd, sealed-secrets) are deployed using ansible kubernetes.core.helm tasks, and are much more difficult to automate deployment. Renovate can update the git repo containing the playbooks (with difficulty, as I do not think Renovate currently supports the kubernetes.core.helm format), but automatically running an ansible playbook (with sudo credentials) afterwards is currently beyond our automation. It is especially here that we have issues with the release cadence. Our current update check for this part consists of a simple in-cluster bash script looking up which charts are installed and looking up updates in a fixed list of repositories, using the helm CLI.

We started with the 2nd setup and only later discovered a better GitOps workflow.

Better release notes would certainly help us here to determine whether an update is worth doing for our ansible playbooks. That still means reviewing quite a few release notes on a weekly basis or so to see if there are security-related updates, but it would make a difference. Right now our process consists mostly of 'open the history of the relevant Chart.yaml file and see if there are relevant changes', which is a lot of manual browsing. This is necessary as security updates for the container/app itself are often patch updates.

I'm also glad to see there is ongoing work to update Go apps to scratch containers.

We are trying to keep our own YAML to a minimum, with a few rules:

• if a project contains CRDs, they need to come from upstream
• if a project contains complex RBAC or many separate statefulsets/deployments/daemonsets, there is strong preference for a chart/kustomize from upstream

This leaves simpler apps like oauth2-proxy and the like to be written in our own YAMLs. I just tried migrating that one, and we ended up with fewer lines of YAML with our own kustomize base and a few overlays compared to actually templating the yaml.