Improve `kong/gateway-operator` Helm chart RBACs and CRDs propagation

[czeslavo]

Problem Statement

Currently, a Helm chart for the operator (kong/gateway-operator) resides in a separate repository (kong/charts), a central place for our charts to live.

The tests verifying an upgrade procedure live in the KGO repository and rely on the Helm chart.

Testing our Helm chart with the main version of the operator comes with an issue - when adding new RBAC rules or CRDs, those have to be updated manually in the charts repo first and released. That makes the process of robust testing before the actual release cumbersome.

This issue aims to find a solution that will make this process smoother, ticking as many as possible of the following boxes:

• kong/gateway-operator chart doesn't have to be released to test new RBACs and CRDs with the chart (that are not yet needed or handled by KGO)
• we do not patch any resources manually in KGO upgrade tests to keep as close as possible to the charts upgrade flow
• auto-generated CRDs and RBACs manifests from KGO are auto-propagated to charts somehow
• charts do not install unnecessary rules/CRDs when using older KGO versions with a newer chart version

Proposed Solutions

Solution A - keep a separate branch in the charts repo

Maintain a kgo/next (or a specific kgo/vX.Y.Z) branch in the kong/charts repository where we'll stack all the changes needed for the next release (and merge it after a KGO release). Test against it in the KGO upgrade tests (could be done by cloning the charts repo and using it locally to avoid unnecessary releases polluting charts).

Pros:

• the charts repository's main branch is not polluted with KGO's unreleased changes (new RBACs, CRDs that are not yet needed and/or handled)
• we can keep releasing from the chart's main (fixes, KGO release cycle-independent features, etc.)

Cons:

• we need to keep two branches instead of one for charts (main and the next release)
• we need to manually update templates on the release branch
• we need to make sure that the next branch is kept in sync with main

Solution B - Make CRDs and RBACs templates conditionally render new content based on KGO versions

Similarly to KIC, introduce version-based conditionals in templates, adding RBAC rules and CRDs if they're satisfied. (Note: this potentially could be automated with a generator iterating over all KGO versions, assembling the output files with conditionals).

This approach would impose the following flow when adding new CRDs/RBACs to KGO and implementing a reconciler for them:

• Add CRD types and kubebuilder markers for RBAC rules in KGO. Merge those to the main branch.
• Wait for automation to regenerate templates in the charts repo based on the updated KGO's main state. Review it and merge it to main.
• Wait for KGO automation to pick up the latest charts' main reference to be used as a dependency in tests.

Pros:

• we have an automated process that keeps KGO and Helm chart's RBACs and CRDs in sync (with eventual consistency)
• we do not need a separate branch in charts to be created for each KGO release, we stick to the main-based process
• we can keep releasing from the chart's main (fixes, KGO release cycle-independent features, etc.)

Cons:

• we still end up with a kind of circular dependency situation where changes in KGO (adding RBACs and CRDs) trigger changes in charts (regenerating RBACs and CRDs based on KGO's source of truth) that in turn trigger changes in charts (dependency update)

Acceptance Criteria

• [ ] A decision about the approach of how we'd like to improve KGO and charts interoperability is made
• [ ] An issue tracking the implementation is created

[pmalek]

Another approach could be

Solution C

Move KGO chart related code into KGO repository. This would allow some parts of the mechanism to be automated, e.g. generating the manager role's policy rules and will resolve the chicken and egg problem that we have.

The chart would still be available to users from https://github.com/Kong/charts/: we'd introduce a mechanism for syncing the chart with https://github.com/Kong/charts/blob/gh-pages/index.yaml which is the source of truth for helm artifacts.

There are 2 challenges that I can see at this moment:

• Using this chart in KGO EE's CI would require some more work. We could leverage the go mod cache path for using the locally stored helm chart directory
• This would require moving some of the machinery that's used for testing charts from https://github.com/Kong/charts/ and hooking that into KGO's CI