Allow configuring multiple Gateways with heterogenous versions/dbmodes

[czeslavo]

Is there an existing issue for this?

• [X] I have searched the existing issues

Problem Statement

702 has introduced a basic functionality allowing configuring multiple Gateways with a single KIC instance. It also introduced a limitation that allows only Gateways with a homogenous versions/dbmodes, which is enforced by a validation rule.

The goal of this issue is to allow KIC to configure Gateways of different versions and dbmodes, effectively dropping the aforementioned validation rule.

Proposed Solution

• Remove the validation rule enforcing Gateways with uniform version/dbmodes across Gateways configured by KIC
• Align code that relies on the Kong Gateway version to account for every Gateway separately
• Possibly introduce some kind of a compatibility matrix that will ensure that we do not end up with a combination of Gateways that won't be able to work as expected in pair (e.g. using some feature introduced in GW's v. X won't be possible to be used when running together with a GW v. X-1).

Additional information

No response

Acceptance Criteria

• [ ] The strict validation rule that doesn't allow heterogenous versions/dbmodes of configured Gateways is removed
• [ ] All the code responsible for configuring Gateways that depend on a version/dbmode properly handles every Gateway according to its version/dbmode
• [ ] Cross-Gateways version compatibility validation rule is introduced to ensure that all Gateways end up with the same configuration (in terms of handling data-plane traffic).

[mflendrich]

Another problem this causes (see #3363): we may end up with different plugins schemas across Gateways

[mheap]

@mflendrich Should we bump this to 2.11?

[mflendrich]

Let's reconsider a blue-green approach to upgrading now that we're working on BG deployments for KGO (@pmalek)

[pmalek]

Let's reconsider a blue-green approach to upgrading now that we're working on BG deployments for KGO (@pmalek)

I might not be seeing the full context here but this is tracks the effort of relaxing the homogeneity requirement and possibly introducing a version/dbmode checks (think cross compatibility) whereas blue-green in KGO would

• be implemented on a different level
• not enforce versions/dbmodes etc

The latter point might be touching on "promotion" custom checks extension of BG at some point but that might be quite elaborate and to be defined using CRD fields (which are not in place yet) instead of relying on a mechanism embedded in KIC.

Having said that: this might be relevant for BG in KGO (https://github.com/Kong/gateway-operator/issues/706) but it all depends on the implementation of it (e.g. if we go the route of using "preview" Service and ReplicaSet as described in the doc then KIC would manage homogenous Gateways anyway).

[programmer04]

Currently, there is a disparity between the requirement of homogeneity during startup https://github.com/Kong/kubernetes-ingress-controller/blob/84a345d92f209e4cb07a1bb965969fe279f7e800/internal/manager/utils/kongconfig/root.go#L33-L70 enforced with https://github.com/Kong/kubernetes-ingress-controller/blob/84a345d92f209e4cb07a1bb965969fe279f7e800/internal/manager/utils/kongconfig/root.go#L33-L70 but later available Kong Gateways (clients for them) are managed with https://github.com/Kong/kubernetes-ingress-controller/blob/84a345d92f209e4cb07a1bb965969fe279f7e800/internal/manager/run.go#L150-L163 and gateway discovery mechanism enforces nothing in terms of homogeneity