kubeadm分析

[llhuii]

kubeadm init phase

https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#kubeadm-init-workflow-internal-design

preflight 检查

certs 生成证书

1. ca：k8s的根证书
2. apiserver：用于apiserver自身启动
3. apiserver-kubelet-client：用于apiserver连接kubelet的证书
4. etcd-ca：etcd的根证书
5. etcd-server： etcd server的根证书
6. etcd-peer：连接其他etcd节点的证书
7. apiserver-etcd-client：apiserver连接etcd的证书
8. sa：给serviceaccount签名
9. front-proxy-ca/front-proxy-client：代理

kubeconfig 生成kubeconfig配置文件

1. admin：管理员使用
2. kubelet：
3. controller-mananger：
4. scheduler：

kubelet-start: 生成kubelet配置和启动kubelet

control-plane: 生成控制面的static pod manifest

1. apiserver
2. controller-mananger：
3. scheduler

etcd: 生成etcd的pod manifest

upload-config: 将kubeadm/kubelet 配置写成configmap

upload-certs: 将ca/etcd-ca/sa 私钥和公钥写到secret kubeadm-certs

mark-control-plane

bootstrap-token

addon 安装格外插件

1. coredns
2. kube-proxy

[llhuii]

bootstrap 启动引导

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/cluster-lifecycle/bootstrap-discovery.md

---

首先以insecure方式system:anonymous访问apiserver，获取到ca证书, 再加上bootstrap-token生成/etc/kubernetes/bootstrap-kubelet.conf

# kubectl get -n kube-public cm cluster-info  -o json | jq .data.kubeconfig -r
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <base64 encoded ca public data>==
    server: https://sedna-mini-control-plane:6443
  name: ""
contexts: null
current-context: ""
kind: Config
preferences: {}
users: null

$ kubectl get -n kube-public rolebinding kubeadm:bootstrap-signer-clusterinfo -oyaml
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubeadm:bootstrap-signer-clusterinfo
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: system:anonymous

# kubectl get -n kube-public role kubeadm:bootstrap-signer-clusterinfo  -o yaml | yq eval .rules -
- apiGroups:
    - ""
  resourceNames:
    - cluster-info
  resources:
    - configmaps
  verbs:
    - get

/etc/kubernetes/bootstrap-kubelet.conf文件内容为：

apiVersion: v1 
clusters:
- cluster:
    certificate-authority-data: LS<根证书base64>==
    server: https://kind-external-load-balancer:6443
  name: kind
contexts:
- context:
    cluster: kind
    user: tls-bootstrap-token-user
  name: tls-bootstrap-token-user@kubernetes
current-context: tls-bootstrap-token-user@kubernetes
kind: Config
preferences: {}
users:
- name: tls-bootstrap-token-user
  user:
    token: qxsijr.4jr8f6o6e41whvu5

---

其次将bootstrap token访问apiserver，被识别group为system:bootstrappers:kubeadm:default-node-token的用户

https://kubernetes.io/zh/docs/reference/access-authn-authz/bootstrap-tokens/#bootstrap-token-secret-format

# kubectl get secret -n kube-system bootstrap-token-abcdef -o json | jq .data | tr -d '",' | while read k v; do echo $k $(echo $v | base64 -d); done
{
auth-extra-groups: system:bootstrappers:kubeadm:default-node-token
expiration: 2021-10-10T06:46:30Z
token-id: abcdef
token-secret: 0123456789abcdef
usage-bootstrap-authentication: true
usage-bootstrap-signing: true
}

---

第三，创建csr请求，为公钥+标识名称签名生成公钥证书。 csr: 允许group为system:bootstrappers:kubeadm:default-node-token创建csr请求

$ kubectl get clusterrole system:certificates.k8s.io:certificatesigningrequests:nodeclient  -o yaml
rules:
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests/nodeclient
  verbs:
  - create

$ kubectl get clusterrolebinding kubeadm:node-autoapprove-bootstrap -o yaml
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:certificates.k8s.io:certificatesigningrequests:nodeclient
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

csr样例：

spec:                                                                                                        
  groups:                                                                                                    
  - system:nodes                          
  - system:authenticated                                                              
  request: LS0t<omit>S0tLQo=
  signerName: kubernetes.io/kube-apiserver-client-kubelet
  usages:
  - digital signature
  - key encipherment
  - client auth
  username: system:node:sedna-mini-control-plane
status:
  certificate: LS0tLS1CR<omit>EUtLS0tLQo=
  conditions:
  - lastTransitionTime: "2021-10-09T15:37:34Z"
    lastUpdateTime: "2021-10-09T15:37:34Z"
    message: Auto approving self kubelet client certificate after SubjectAccessReview.
    reason: AutoApproved
    status: "True"
    type: Approved

status字段由csrapprover controller-manager监听csr并更新。见https://code.k8s.io/pkg/controller/certificates/approver/sarapprove.go#L126

upload-certs

$ kubectl get -n kube-system rolebinding kubeadm:kubeadm-certs -o yaml

roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubeadm:kubeadm-certs
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:bootstrappers:kubeadm:default-node-token

$ kubectl get -n kube-system role kubeadm:kubeadm-certs -o yaml
rules:
- apiGroups:
  - ""
  resourceNames:
  - kubeadm-certs
  resources:
  - secrets
  verbs:
  - get

[llhuii]

bootstrap 启动引导（接上）

查看csr的证书信息O = system:nodes, CN = system:node:sedna-mini-control-plane

# kubectl get csr csr-xb6jj -o json | jq .status.certificate -r |base64  -d | openssl x509 -noout  -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            8e:9a:fa:c8:fa:31:01:e9:5e:e9:0c:c4:80:d4:94:d0
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = kubernetes
        Validity
            Not Before: Oct  9 15:32:34 2021 GMT
            Not After : Oct  9 15:32:34 2022 GMT
        Subject: O = system:nodes, CN = system:node:sedna-mini-control-plane