decide what to do with CNI, and also should clusters have more than one profile?

[bcle]

Now that arlon supports CAPI-AWS-kubeadm clusters, the issue of CNI came up. A CAPI kubeadm cluster starts with no CNI, so any bundles that install workloads (e.g. applications) will be stuck in the progressing state. This was a great opportunity for me to define a CNI setup as a bundle (I created a static bundle from a file downloaded from https://docs.projectcalico.org/v3.21/manifests/calico.yaml), and then add the bundle to the cluster's dynamic profile. As expected, the cluster was automatically updated with calico, and the workloads eventually deployed successfully. However, if CAPI EKS clusters were using the same dynamic profile, they would potentially break, since they are automatically configured with AWS VPC CNI out of the box, resulting in a conflict. So this raises important questions about how to treat CNIs. Possible approaches:

• Since CNIs are installable as bundles, keep them separate from cluster specs. Profiles should be defined with CNI bundles for clusters that have no OOB CNI, and separate profiles with no CNI bundle should be used for clusters that have CNI OOB. The problem with this is unnecessary duplication and proliferation of redundant profiles that only differ by one bundle.
• Make CNI a property of a cluster spec. This solves the profile duplication problem, but it means that CNIs (which again, are installable bundles) need special treatment by arlon, as opposed to managed by the user. For example, arlon would need to manage CNI releases, and test compatibility matrices.
• Allow clusters to have more than one profile. This would allow users to organize profiles into e.g. "infrastructure", "add-on", "config", and "apps", thereby eliminating duplication by allowing a cluster to use the exact set of profiles that it needs. The CNI bunlde could then be configured as part of a particular "infrastructure" profile, and that profile only needs to be included when absolutely needed by the cluster.

Aha! Link: https://pf9.aha.io/features/ARLON-169

[cre8minus1]

@bcle what was the verdict on CNI? and should the Multiple Profile issue be split out