olga-mir
commented
2 years ago Current status snapshot
Permanent management cluster
Flux and general state
% k config use-context mgmt
Switched to context "mgmt".
%
% flux get all
NAME REVISION SUSPENDED READY MESSAGE
gitrepository/flux-system develop/6295448 False True stored artifact for revision 'develop/62954486768a0a018a1ec533e7da82961d0e636e'
NAME REVISION SUSPENDED READY MESSAGE
helmrepository/kubefed 4aebe12802a1d4cf3e81c3ea755fa6a45e63fcc06f6fbfdb0826b6c82be67265 False True stored artifact for revision '4aebe12802a1d4cf3e81c3ea755fa6a45e63fcc06f6fbfdb0826b6c82be67265'
NAME REVISION SUSPENDED READY MESSAGE
helmchart/kube-federation-system-kubefed 0.9.2 False True pulled 'kubefed' chart with version '0.9.2'
NAME REVISION SUSPENDED READY MESSAGE
kustomization/flux-system develop/6295448 False True Applied revision: develop/6295448
kustomization/infrastructure develop/6295448 False True Applied revision: develop/6295448
%
% k get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
capa-system capa-controller-manager-7956c788f5-nswzh 1/1 Running 0 18m
capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-88bf6c77f-wl8ct 1/1 Running 0 18m
capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-7c99cb7ffd-88dzj 1/1 Running 0 18m
capi-system capi-controller-manager-74fdc5ff64-6w8gw 1/1 Running 0 18m
cert-manager cert-manager-6bb4c6d67d-kx9b2 1/1 Running 0 20m
cert-manager cert-manager-cainjector-5779577c5f-26k94 1/1 Running 0 20m
cert-manager cert-manager-webhook-5fbbdccffd-8qwgp 1/1 Running 0 20m
flux-system helm-controller-747fb76c85-2pnk5 1/1 Running 0 17m
flux-system kustomize-controller-666b6f4769-xhpv6 1/1 Running 0 17m
flux-system notification-controller-57cd4f4df-pwgrw 1/1 Running 0 21m
flux-system source-controller-d594d944c-mbxrt 1/1 Running 0 17m
kube-federation-system kubefed-admission-webhook-5b9d78b747-7bh8k 1/1 Running 0 2m47s
kube-federation-system kubefed-controller-manager-864bc86d6b-2kvgr 1/1 Running 0 2m4s
kube-federation-system kubefed-controller-manager-864bc86d6b-88zf8 1/1 Running 0 2m6s
kube-system cilium-58w6m 1/1 Running 0 21m
kube-system cilium-operator-74c447879b-2l6ss 1/1 Running 0 21m
kube-system cilium-operator-74c447879b-xbmpn 1/1 Running 0 21m
kube-system cilium-ztwdm 1/1 Running 0 20m
kube-system coredns-64897985d-gkdcj 1/1 Running 0 21m
kube-system coredns-64897985d-grnz2 1/1 Running 0 21m
kube-system etcd-ip-10-0-192-58.ap-southeast-2.compute.internal 1/1 Running 0 21m
kube-system kube-apiserver-ip-10-0-192-58.ap-southeast-2.compute.internal 1/1 Running 0 21m
kube-system kube-controller-manager-ip-10-0-192-58.ap-southeast-2.compute.internal 1/1 Running 0 21m
kube-system kube-scheduler-ip-10-0-192-58.ap-southeast-2.compute.internal 1/1 Running 0 21mCluster API resources:
% k get clusters -A
NAMESPACE NAME PHASE AGE VERSION
cluster-dev dev Provisioned 31m
cluster-mgmt mgmt Provisioned 31m
%
% k get kcp -A
NAMESPACE NAME CLUSTER INITIALIZED API SERVER AVAILABLE REPLICAS READY UPDATED UNAVAILABLE AGE VERSION
cluster-dev dev-control-plane dev true true 1 1 1 0 31m v1.21.11
cluster-mgmt mgmt-control-plane mgmt true true 1 1 1 31m v1.23.8
%
% k get providers -A
NAMESPACE NAME AGE TYPE PROVIDER VERSION
capa-system infrastructure-aws 32m InfrastructureProvider aws v1.4.1
capi-kubeadm-bootstrap-system bootstrap-kubeadm 32m BootstrapProvider kubeadm v1.2.0-beta.0
capi-kubeadm-control-plane-system control-plane-kubeadm 32m ControlPlaneProvider kubeadm v1.2.0-beta.0
capi-system cluster-api 32m CoreProvider cluster-api v1.2.0-beta.0Kubernetes federation
% k get kfc -A
NAMESPACE NAME AGE READY KUBERNETES-VERSION
kube-federation-system dev 105s True v1.21.11
kube-federation-system mgmt 105s True v1.23.8Workload cluster
Not sure why kube-proxy isntall was not skipped. It has the same setting in kcp and it was correctly applied to the cluster.
Flux and general state
% k get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
flux-system helm-controller-86d8f4495c-xknkw 1/1 Running 0 3m14s
flux-system kustomize-controller-6c8f5445d9-725p4 1/1 Running 0 3m14s
flux-system notification-controller-6b7d7485fc-brc9s 1/1 Running 0 21m
flux-system source-controller-57745f55f-kh2wl 1/1 Running 0 3m12s
kong kong-kong-6c746dfcfb-jj5z7 2/2 Running 2 2m39s
kube-system cilium-j7w8n 1/1 Running 0 8m13s
kube-system cilium-operator-7d56c6f4db-8tdv9 1/1 Running 0 8m13s
kube-system cilium-operator-7d56c6f4db-k2hz9 1/1 Running 0 8m13s
kube-system cilium-wjnx8 1/1 Running 0 8m13s
kube-system coredns-558bd4d5db-khqmq 1/1 Running 0 21m
kube-system coredns-558bd4d5db-r74mz 1/1 Running 0 21m
kube-system etcd-ip-10-0-156-220.ap-southeast-2.compute.internal 1/1 Running 0 21m
kube-system kube-apiserver-ip-10-0-156-220.ap-southeast-2.compute.internal 1/1 Running 0 21m
kube-system kube-controller-manager-ip-10-0-156-220.ap-southeast-2.compute.internal 1/1 Running 0 21m
kube-system kube-proxy-4mdrc 1/1 Running 0 21m
kube-system kube-proxy-hqwqd 1/1 Running 0 20m
kube-system kube-scheduler-ip-10-0-156-220.ap-southeast-2.compute.internal 1/1 Running 0 21m
%
%
% flux get all
NAME REVISION SUSPENDED READY MESSAGE
gitrepository/flux-system develop/6295448 False True stored artifact for revision 'develop/62954486768a0a018a1ec533e7da82961d0e636e'
NAME REVISION SUSPENDED READY MESSAGE
helmrepository/kong a7b6cbbee2d2e0faf81d55ac1fc62c7301fbed62fbcd25035555327090186138 False True stored artifact for revision 'a7b6cbbee2d2e0faf81d55ac1fc62c7301fbed62fbcd25035555327090186138'
NAME REVISION SUSPENDED READY MESSAGE
helmchart/kong-kong 2.8.2 False True pulled 'kong' chart with version '2.8.2'
NAME REVISION SUSPENDED READY MESSAGE
kustomization/flux-system develop/6295448 False True Applied revision: develop/6295448
kustomization/infrastructure develop/6295448 False True Applied revision: develop/6295448
kustomization/tenants develop/6295448 False True Applied revision: develop/6295448
UPD: https://github.com/olga-mir/k8s-multi-cluster/pull/8#issuecomment-1169814895
This PR is trying to move as much of the operations/config as possible in GitOps declarative manner by installing flux on tmp management cluster and utilising more ClusterResourceSets on all clusters (except the temp one) to install CNI and flux. More details can be found in this PR in
docsfolder. Along the way Calico has been swapped in favour of Cilium in order to prepare for cluster mesh. However this setup did not work as expected for a few reasons: Cilium can work in kube-proxy-free way or in partial coexistence. CABPK currently doesn't supportskipPhasesflag to disable proxy installation (it is implemented but targeted in v1.2). For the purpose of this project it is for time being to manually remove kube-proxy. However, this doesn't work in CRS approach because Cilium needs to know the actual API server host, which at the time of cluster bootstrap is not known. It seems that Cilium can live with this set to API server LB, in which case it can be possible by BYO infra (so that ELB name is known before cluster creation). The BYO infra doc doesn't recommend to BYO ELB though. I have tried a few configs for Cilium chart to make it work with or without kube-proxy (and without explicitk8sServiceHost) but they were broken in different ways. Images cannot be pulled (coreDNS logs timeoutsread udp 10.0.0.162:44756->10.0.0.2:53: i/o timeout, registry URLs can't be resolved:failed to do request: Head "https://quay.io/v2/jetstack/cert-manager-controller/manifests/v1.5.3": dial tcp: lookup quay.io: Temporary failure in name resolution) Turns out this is due to not providing explicit pod CIDR mask to cilium, the default that cilium assigned to the podcidr is "10.0.0.0/8" and therefore from cilium perspective "10.0.0.2" is in pod space, but there is no such pod, and packets destined to this IP were dropped: