search

kubernetes-sigs / kind

Kubernetes IN Docker - local clusters for testing Kubernetes
https://kind.sigs.k8s.io/
Apache License 2.0
13.5k stars 1.56k forks source link

kubeadm clusterconfiguration apiServer.extraArgs overwrite issue #1839

Open bjethwan opened 4 years ago

bjethwan commented 4 years ago

kind version kind v0.8.1 go1.14.2 darwin/amd64

kind create cluster --config - <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes: 
- role: control-plane
  extraMounts:
  - containerPath: /etc/kubernetes/oidc
    hostPath: /tmp/oidc
- role: worker
- role: worker
- role: worker
kubeadmConfigPatches:
- |
  apiVersion: kubeadm.k8s.io/v1beta2
  kind: ClusterConfiguration
  apiServer:
    extraArgs:
      api-audiences: "sts.amazonaws.com"
      service-account-key-file: "/etc/kubernetes/pki/sa.pub"
      service-account-key-file: "/etc/kubernetes/oidc/sa-signer-pkcs8.pub"
      service-account-signing-key-file: "/etc/kubernetes/oidc/sa-signer.key"
      service-account-issuer: "https://s3-XXXXXXXXXX.amazonaws.com/XXXXXXXXXXXXXXXXXXXXX"
    extraVolumes:
    - name: "oidc"
      hostPath: "/etc/kubernetes/oidc"
      mountPath: "/etc/kubernetes/oidc"
EOF

Issue: Nodes stay in NotReady state indefinitely.

k get nodes
NAME                 STATUS     ROLES    AGE     VERSION
kind-control-plane   NotReady   master   8m7s    v1.18.2
kind-worker          NotReady   <none>   7m40s   v1.18.2
kind-worker2         NotReady   <none>   7m33s   v1.18.2
kind-worker3         NotReady   <none>   7m33s   v1.18.2

Kind/kubeadm missed to write "service-account-key-file", when k8s docs state that it can appear multiple times in kube-apiserver command line args.

spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=172.18.0.5
    - --allow-privileged=true
    - --api-audiences=sts.amazonaws.com
    - --authorization-mode=Node,RBAC
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --enable-admission-plugins=NodeRestriction
    - --enable-bootstrap-token-auth=true
    - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt
    - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt
    - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key
    - --etcd-servers=https://127.0.0.1:2379
    - --insecure-port=0
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --requestheader-allowed-names=front-proxy-client
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-username-headers=X-Remote-User
    - --secure-port=6443
    - --service-account-issuer=https://s3-ap-south-1.amazonaws.com/XXXXXXXXXXXXXXXXXXXXXXXX
    - --service-account-key-file=/etc/kubernetes/oidc/sa-signer-pkcs8.pub
    - --service-account-signing-key-file=/etc/kubernetes/oidc/sa-signer.key
    - --service-cluster-ip-range=10.96.0.0/12
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key

Everything starts working when I do this step manually docker exec -it kind-control-plane bash root@kind-control-plane:/# vi /etc/kubernetes/manifests/kube-apiserver.yaml

Update this line - --service-account-key-file=/etc/kubernetes/pki/sa.pub

exit

$ k get nodes
NAME                 STATUS   ROLES    AGE     VERSION
kind-control-plane   Ready    master   9m35s   v1.18.2
kind-worker          Ready    <none>   9m8s    v1.18.2
kind-worker2         Ready    <none>   9m1s    v1.18.2
kind-worker3         Ready    <none>   9m1s    v1.18.2
neolit123 commented 4 years ago

See https://github.com/kubernetes/kubeadm/issues/1601 This is broken in kubeadm and should be fixed in a new api - v1beta3.

Until then, one workaround for you is to go in the kind nodes and patch the manifest manually to add the flag that kubeadm omitted.

irbekrm commented 2 years ago

I also noticed that extra args for etcd do not work in v1beta2.

A kind cluster created with the below configuration results in unsafe-no-fsync flag not used. If I change apiVersion to kubeadm.k8s.io/v1beta3 it works.

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
kubeadmConfigPatches:
  - |
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterConfiguration
    metadata:
      name: config
    etcd:
      local:
        extraArgs:
          unsafe-no-fsync: "true"
nodes:
  - role: control-plane

Is this likely related to the same issue in kubeadm?

Would it make sense to add a warning to kind docs that v1beta3 should be used for these patches?

BenTheElder commented 2 years ago

Is this likely related to the same issue in kubeadm?

No, this is just because of the Kubernetes version, kind uses the latest available config version for kubeadm, which varies by kubernetes version

For stable fields, you can write. patch like this:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
kubeadmConfigPatches:
  - |
    kind: ClusterConfiguration
    metadata:
      name: config
    etcd:
      local:
        extraArgs:
          unsafe-no-fsync: "true"
nodes:
  - role: control-plane

(apiVersion omitted from the patch) and kind will permit it and match based on kind: ClusterConfiguration and patch, but using the api version would allow you to do this:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
kubeadmConfigPatches:
  - |
    apiVersion: kubeadm.k8s.io/v1beta3
    kind: ClusterConfiguration
    metadata:
      name: config
    etcd:
      local:
        extraArgs:
          unsafe-no-fsync: "true"
  - |
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterConfiguration
    metadata:
      name: config
    etcd:
      local:
        extraArgs:
          unsafe-no-fsync: "true"
nodes:
  - role: control-plane

(and have different results for different API levels)

EDIT: this issue is about what happens if you try to specify multiple values for a flag

irbekrm commented 2 years ago

Thanks a lot for that explanation @BenTheElder that makes sense! I guess then we might actually skip the apiVersion field for configs that are meant to work for different Kubernetes versions (I realized that setting it to kubeadm.k8s.io/v1beta3 makes it work for 1.24, but not work for 1.21.. whereas without the apiVersion the same config works for both 👍🏼 )

Sorry for hijacking an unrelated issue.