Open bh-tt opened 1 day ago
Hi! First of all thank so much for this detailed feedback! Also thank you for using Bitnami for all these years and having contributed with features to some of our charts. We're definitely sorry to see you go. Let us share our views on the current release process:
To provide some context, this release process of "one release per change" is inherited from the initial, now deprecated, "helm/charts" repository maintained by the official helm team. Bitnami was a core contributor on this repository and, when this was discontinued, we simply followed the same approach to maintain the same UX and expectations to the community.
Having said that, we also believe that making a feature available as soon as possible is extremely helpful to the community. I can think of: Solutions to bugs, HIGH/CRITICAL CVEs fixed, useful features... In addition to this, it is helpful for us as the effort to monitor and decide when to cut releases of more than 100 charts could easily become out of hand.
However, the cases you highlighted (etcd, metrics-server) definitely illustrate a painful point if you want to keep the charts up to date. As you mentioned, most of these Automated Releases (PRs with the title "Release
You definitely have a point on Go apps that could become scratch containers. We tend to use scratch containers whenever it makes sense and if the upstream project uses scratch too. We have on-going initiatives to review our Go apps to see if they can be changed to scratch, so you can expect some solutions to change as long as the upstream project also uses scratch/distroless.
As soon as I saw your post, I immediately thought: "what's preventing them to choose their own release cadence?" That is, for example discard all patch updates and go only for minors, or maybe go for updates that change the appVersion, and so on. If possible, could you share with us the process you followed when a new chart version was released? Something tells me that a major pain point was the lack of proper visibility on the changes (especially those automated). In order to mitigate this, we began using git tags and changelogs so you could easily see the changes between two chart versions. For example:
git log wordpress/19.4.1..wordpress/22.2.1 --oneline bitnami/wordpress
You can also use the changelog with automation tools like renovate or dependabot, making the process easier. However, we believe that we could do better with the changelog entries (especially with the automated ones). I was thinking that, if we had a consistent, standard way of titling the changes, for example: "[bitnami/metrics-server] chore(security): Fix CVE-XXXX-YYYY", "[bitnami/metrics-server] feat: Add PDB support", you could easily track and discard releases, therefore creating your custom update cadence tailored to your needs.
We would love to hear your thoughts on this. Maintaining your own YAML can definitely be painful too, so it would be great to know if changes like the above (improve the visibility of changes and review Go applications) could help you reconsider your decision.
Once again, thank you for your feedback and continued usage through all these years.
We have 2 processes for updates:
helm
CLI. We started with the 2nd setup and only later discovered a better GitOps workflow.
Better release notes would certainly help us here to determine whether an update is worth doing for our ansible playbooks. That still means reviewing quite a few release notes on a weekly basis or so to see if there are security-related updates, but it would make a difference. Right now our process consists mostly of 'open the history of the relevant Chart.yaml file and see if there are relevant changes', which is a lot of manual browsing. This is necessary as security updates for the container/app itself are often patch updates.
I'm also glad to see there is ongoing work to update Go apps to scratch containers.
We are trying to keep our own YAML to a minimum, with a few rules:
This leaves simpler apps like oauth2-proxy and the like to be written in our own YAMLs. I just tried migrating that one, and we ended up with fewer lines of YAML with our own kustomize base and a few overlays compared to actually templating the yaml.
Name and Version
all
What is the problem this feature will solve?
This really belongs in a discussion, but since this project does not have them enabled I'll open a feature request.
I'm from a small IT team managing a medium-sized infrastructure (several clusters, from 6 to 15 nodes large nodes in size). We started migrating to Kubernetes a while back, and bitnami's chart really helped us setup a lot of basic infrastructure, such as metrics-server, an etcd, postgres databases and metric-based alerting. The charts are comprehensive and well-documented, and we added a few features ourselves as well. However, after a few years we are moving away from bitnami charts for most of our deployments, replacing most releases with kustomize bases directly from the deployed application or even writing our own plain YAML files. Why?
Because of the release cadence. For production-critical infrastructure like a postgres database or argo-workflows controller, updating (and suffering downtime and/or update problems) very often is hard. Bitnami releases very often, even for projects that themselves are very stable (e.g. metrics-server, etcd, postgres, with a few to 10 or so releases per year). We found ourselves spending 1-2 hours a week updating bitnami charts, most of which had no application updates and so did not really need an update. To give a few examples:
Why does bitnami release this often? I think there are a few reasons:
FROM scratch
base image, such as most Go applications (half the charts in this repository suffer from this problem). This naturally causes another release for a lot of charts whenever the base image updates, I see 7 updates since debian bookworms' release.I made this issue mostly to create some awareness about the impact of a high release cadence on users, especially for charts that are hard to upgrade without supervision.
What is the feature you are proposing to solve the problem?
I'm not sure there is an easy solution here. Using a helm chart means you'll always have more updates than by simply writing your own YAML files (assuming the helm chart also releases with each application update), but I'd expect a multiplier of 2-3, not 5-15.
There are a few things that can be done however:
What alternatives have you considered?
I'd like to stress this is not a criticism of the charts themselves. Consider this more an alert for something that may cost this repository users in the long term. The charts are very good, well-maintained and have a template variable for everything you can think of. But, when releasing charts for applications that are often critical, it may be good to consider a slower release cadence than 'whenever there is any change'.