Closed olga-mir closed 1 year ago
minimal workflow is working. (permanent cluster-mgmt and 1 workload cluster-01, fully automated)
% kubectx | head cluster-01-admin@cluster-01 cluster-mgmt-admin@cluster-mgmt docker-desktop kind-kind
cluster-mgmt
% kubectx cluster-mgmt-admin@cluster-mgmt
Switched to context "cluster-mgmt-admin@cluster-mgmt".
% k get clusters -A
NAMESPACE NAME PHASE AGE VERSION
cluster-01 cluster-01 Provisioned 9m43s
cluster-mgmt cluster-mgmt Provisioned 12m
k8s-multi-cluster % flux get all -A
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system gitrepository/flux-system develop/a7113c6 False True stored artifact for revision 'develop/a7113c6fe24ab4258d0f97ba5fd50c908fafeb73'
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
cluster-01 kustomization/flux-remote develop/a7113c6 False True Applied revision: develop/a7113c6
flux-system kustomization/caaph develop/a7113c6 False True Applied revision: develop/a7113c6
flux-system kustomization/caaph-cni develop/a7113c6 False True Applied revision: develop/a7113c6
flux-system kustomization/flux-system develop/a7113c6 False True Applied revision: develop/a7113c6
% k get po -A | grep -E "flux-system|cilium"
cilium cilium-6sf7r 1/1 Running 0 31m
cilium cilium-kt6wf 1/1 Running 0 31m
cilium cilium-operator-747b974bf5-5xddz 1/1 Running 0 31m
cilium cilium-operator-747b974bf5-pgls8 1/1 Running 0 31m
cilium hubble-relay-69b687f7c9-znhvc 1/1 Running 0 31m
cilium hubble-ui-5dc4d884b6-gcgm5 2/2 Running 0 31m
flux-system helm-controller-b95c4d69b-qf9pg 1/1 Running 0 17m
flux-system kustomize-controller-5c6c9f7dcc-mwpzd 1/1 Running 0 17m
flux-system notification-controller-5cd6f76fbb-4t69x 1/1 Running 0 17m
flux-system source-controller-84f6bcbfb8-wfn7x 1/1 Running 0 17m
cluster-01
% c cluster-01-admin@cluster-01
Switched to context "cluster-01-admin@cluster-01".
% k get po -A | grep -E "flux-system|cilium"
cilium cilium-9t64n 1/1 Running 0 6m43s
cilium cilium-operator-5d67db599f-px4j9 1/1 Running 0 6m43s
cilium cilium-operator-5d67db599f-v6qr9 1/1 Running 0 6m43s
cilium cilium-vttj7 1/1 Running 0 6m2s
cilium hubble-relay-69b687f7c9-h7pdj 1/1 Running 0 6m43s
cilium hubble-ui-5dc4d884b6-7b5b7 2/2 Running 0 6m43s
flux-system helm-controller-b95c4d69b-g8slz 1/1 Running 0 5m58s
flux-system kustomize-controller-5c6c9f7dcc-4znpq 1/1 Running 0 5m58s
flux-system notification-controller-5cd6f76fbb-vctsz 1/1 Running 0 5m58s
flux-system source-controller-84f6bcbfb8-rnnxx 1/1 Running 0 5m58s
Re-reviewing again approach outlined in this repo https://github.com/fluxcd/flux2-multi-tenancy I think I got it wrong. In my case
clusters
were no more than just pointers to kustomizations ininfrastructure
.infrastructure
in this example as I understand it now is a collection of "lego" blocks, components that run on the clusters (not "payload"), andclusters
is the assembly of the lego blocks. I also findinfrastructure
word a bit confusing in this context and I think thatplatform
is more appropriate.ClusterResourceSet
is an experimental feature and was never meant to stay in CAPI, however currently there is no fully integrated solution to take its place. This is likely to be that solution one day: https://github.com/Jont828/cluster-api-addon-provider-helm This will be used in this repo to install Cilium on workload clusters (including the permanent management cluster). It can't be installed with https://fluxcd.io/flux/components/helm/helmreleases/#remote-clusters--cluster-api because it needs dynamic value (api server host and port) which is only known at the runtime. CAAPH can handle this.Note on cluster index
Cluster index is required for Cilium mesh, but current CAAPH implementation doesn't seem to be able to extract index from the labels.
Moreover, if using labels as a way to pass index there is type mismatch since labels are strings, but Cilium expects it as an integer.
The workaround for now is to duplicate HelmReleaseProxy and hard-code the index :(
id: {{ .ControlPlane.metadata.name | strings.TrimPrefix "cluster-0" }}
failed because "strings" not defined.id: {{ .ControlPlane.metadata.labels.clusterindex }}
this works, but cilium needs it as integer, while labels need to be strings