Utilise CrossPlane to manage Keycloak deployment related cloud infrastructure

[ryanemerson]

Description

Crossplane allows users to provision cloud infrastructure using K8s CRDs. We can greatly simplify our current Taskfiles by replacing the various AWS bash scripts with a Helm based solution that deploys Crossplane CRs.

As Crossplane requires a K8s cluster to exist, the provisioning of ROSA clusters will continue to be OpenTofu based.

Discussion

Pros

• Cloud resources created/deleted on CR removal
• Crossplane is cloud-agnostic, so support for new cloud-vendor simply requires new CRs
• Strong community
• Bash scripts can be removed
• Simpler integration with taskfiles than OpenTofu modules as K8s manages state for us

Cons

• We need to ensure that CRs are removed before cluster removal to ensure cloud resources are cleaned up
• Yet another tool

Alternatives

Terraform/OpenTofu could also be used for this purpose, however there are drawbacks to such an approach:

• State files need to be maintained on S3 to allow distributed working
• Workspace required per deployment of same resource types
• Terraform K8s integration has limitations as outlined here

[ryanemerson]

A challenge with Crossplane is how to manage cloud state that is relied upon by a deployment that exists on multiple K8s clusters, e.g. Aurora clusters for Active/Passive deployments.

It seems like many people solve this problem by having a dedicated K8s cluster that acts as a control-plane for all other K8 clusters. This seems overkill for our needs.

An alternative solution is to deploy resources on Site A with Crossplane CRs and then utilise Observe Only Resources on Site B to allow retrieval of resource config and state.