search

kinvolk / lokomotive

🪦 DISCONTINUED Further Lokomotive development has been discontinued. Lokomotive is a 100% open-source, easy to use and secure Kubernetes distribution from the volks at Kinvolk
https://kinvolk.io/lokomotive-kubernetes/
Apache License 2.0
321 stars 49 forks source link

Make Cluster API management work out of the box on Lokomotive #1265

Open invidian opened 3 years ago

invidian commented 3 years ago

Right now running clusterctl init --infrastructure packet on fresh Lokomotive cluster on AWS ends up with the following situation:

$ kgpoall
+ kubectl get pods --all-namespaces
NAMESPACE                            NAME                                                              READY   STATUS                       RESTARTS   AGE
capi-kubeadm-bootstrap-system        capi-kubeadm-bootstrap-controller-manager-7ffb7c9d77-5lhkn        0/2     CreateContainerConfigError   0          2m2s
capi-kubeadm-control-plane-system    capi-kubeadm-control-plane-controller-manager-5b8cf46bb6-7bkm6    0/2     CreateContainerConfigError   0          118s
capi-system                          capi-controller-manager-559db48f6-4xzl6                           0/2     CreateContainerConfigError   0          2m7s
capi-webhook-system                  capi-controller-manager-76d9b5889c-zjck2                          0/2     CreateContainerConfigError   0          2m12s
capi-webhook-system                  capi-kubeadm-bootstrap-controller-manager-787cb85f58-6wpgd        0/2     CreateContainerConfigError   0          2m5s
capi-webhook-system                  capi-kubeadm-control-plane-controller-manager-86c44777c5-cmh76    0/2     CreateContainerConfigError   0          2m1s
cert-manager                         cert-manager-cainjector-fc6c787db-4xblv                           1/1     Running                      0          2m37s
cert-manager                         cert-manager-d994d94d7-k4s9j                                      1/1     Running                      0          2m37s
cert-manager                         cert-manager-webhook-845d9df8bf-bdgw4                             1/1     Running                      0          2m36s
cluster-api-provider-packet-system   cluster-api-provider-packet-controller-manager-7c657bc8d8-smgb4   0/2     CreateContainerConfigError   0          112s
kube-system                          calico-kube-controllers-855c8775f9-fkcmz                          1/1     Running                      0          17m
kube-system                          calico-node-pz57t                                                 1/1     Running                      0          16m
kube-system                          calico-node-qhmdq                                                 1/1     Running                      0          16m
kube-system                          coredns-7d799bc4c8-b5tz9                                          1/1     Running                      0          17m
kube-system                          kube-apiserver-7ffd9f7d88-wpcdw                                   1/1     Running                      0          17m
kube-system                          kube-controller-manager-f794b896d-fmw5x                           1/1     Running                      0          17m
kube-system                          kube-proxy-68b7j                                                  1/1     Running                      0          16m
kube-system                          kube-proxy-p6lgq                                                  1/1     Running                      0          16m
kube-system                          kube-scheduler-845bddddfd-8dt9b                                   1/1     Running                      0          17m
kube-system                          kubelet-c9ccg                                                     1/1     Running                      0          16m
kube-system                          kubelet-fnkhh                                                     1/1     Running                      0          16m
kube-system                          pod-checkpointer-sgn2t                                            1/1     Running                      0          15m
kube-system                          pod-checkpointer-sgn2t-ip-10-0-2-61                               1/1     Running                      0          15m
lokomotive-system                    admission-webhook-server-64859d4f48-7lrxm                         1/1     Running                      0          17m

This is because of PSPs we ship and https://github.com/kubernetes-sigs/cluster-api/issues/3836.

PSPs can be workaround by applying the following manifests:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: privileged-psp-capt
  namespace: capi-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:capi-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: privileged-psp-capt
  namespace: capi-kubeadm-bootstrap-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:capi-kubeadm-bootstrap-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: privileged-psp-capt
  namespace: capi-kubeadm-control-plane-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:capi-kubeadm-control-plane-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: privileged-psp-capt
  namespace: capi-kubeadm-control-plane-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:capi-kubeadm-control-plane-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: privileged-psp-capt
  namespace: capi-webhook-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:capi-webhook-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: privileged-psp-capt
  namespace: cert-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:cert-manager
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: privileged-psp-capt
  namespace: cluster-api-provider-packet-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: privileged-psp
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:serviceaccounts:cluster-api-provider-packet-system

This makes pod spawn, but they are still crashing with the logs like:

$ klo cluster-api-provider-packet-controller-manager-7c657bc8d8-p2p4p -c manager
+ kubectl logs -f cluster-api-provider-packet-controller-manager-7c657bc8d8-p2p4p -c manager
2020-12-11T09:08:36.662Z        ERROR   controller-runtime.client.config        unable to get kubeconfig        {"error": "invalid configuration: no configuration has been provided", "errorCauses": [{"error": "no configuration has been provided"}]}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/client/config.GetConfigOrDie
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.5.11/pkg/client/config/config.go:159
main.main
        /workspace/main.go:116
runtime.main
        /usr/local/go/src/runtime/proc.go:203
invidian commented 3 years ago

Hmm, this turns out to be caused by #669.

Applying this extra manifests and re-creating all pods make things work:

apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  name: default
  namespace: capi-system
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  name: default
  namespace: capi-kubeadm-bootstrap-system
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  name: default
  namespace: capi-kubeadm-control-plane-system
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  name: default
  namespace: capi-webhook-system
---
apiVersion: v1
automountServiceAccountToken: true
kind: ServiceAccount
metadata:
  name: default
  namespace: cluster-api-provider-packet-system

I guess other than documentation there is not much we can do with that, except of course contributing and solving kubernetes-sigs/cluster-api#3836.