[k8s-keystone-auth] kubeadm init phase fails when webhook not ready

[heytrav]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug /kind feature

What happened:

kubeadm is unable to bootstrap the admin user when --authorization-mode has Webhook at cluster init.

For background, we're running a ClusterAPI-based platform for deplying clusters and have been using the following configuration to set up the kube-apiserver to use the k8s-keystone-auth webhook.

  clusterConfiguration:
    apiServer:
      extraArgs:
        cloud-provider: external
        authorization-mode: Node,Webhook,RBAC
        authentication-token-webhook-config-file: /etc/kubernetes/webhooks/keystone_webhook_config.yaml
        authorization-webhook-config-file: /etc/kubernetes/webhooks/keystone_webhook_config.yaml

This is applied when kubeadm init runs and is bootstrapping the control plane and has the net effect of creating the /etc/kubernetes/manifests/kube-apiserver.yaml with the necessary arguments for the auth webhook.

Once the control plane has been bootstrapped we are using a Helm chart to deploy k8s-keystone-auth similar to the example code. Up until Kubernetes v1.28.9 this has worked fine even though the webhook initially can't reach the k8s-keystone-auth Pod.

In v1.29 however kubeadm init fails, presumably because the webhook is not responding and also because the admin user is no longer in system:masters and can not authenticate (See https://github.com/kubernetes/kubernetes/pull/121305).

Apr 10 02:00:38 gitlab-1-29-3-42132-nv6dtqdoo7wv-control-plane-63f90477-qjv27 kubeadm.sh[1666]: error execution phase upload-config/kubeadm: could not bootstrap the admin user in file admin.conf: unable to create ClusterRoleBinding: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

What you expected to happen:

How to reproduce it:

Create a v1.29 cluster with the kube-apiserver arguments when kubeadm init first initialises control plane:

     -  --authorization-mode: Node,Webhook,RBAC
     -  --authentication-token-webhook-config-file: /etc/kubernetes/webhooks/keystone_webhook_config.yaml
     -  --authorization-webhook-config-file: /etc/kubernetes/webhooks/keystone_webhook_config.yaml

Anything else we need to know?:

I have a work around in our ClusterAPI setup in which I do not add the --authorization-* arguments until after kubeadm init run. Once kubeadm init has finished I run kustomize to add the arguments to the kube-apiserver.yaml manifest.

It seems to work, but we're concerned about a potential race condition and also wondering if there is a cleaner approach.

Your docs suggest using static pods, however I am not sure how to set up the needed ServiceAccount, ClusterRoleBindings, etc. It seems like the pod would also just not run until that is in place. So far my attempts to use a static pod at cluster init have also failed.

Environment:

• Kubernetes v1.29.3
• k8s-keystone-auth: v1.29.0

[dulek]

Well, sounds like static pod is the way to go, but as we can read in docs:

Note: The spec of a static Pod cannot refer to other API objects (e.g., ServiceAccount, ConfigMap, Secret, etc).

I see that SA is only needed to fetch the ConfigMap from K8s API. I think that you can use --keystone-policy-file instead of --policy-configmap-name and just feed this file directly to the pod as hostPath? Same should be done with certs.

[heytrav]

Well, sounds like static pod is the way to go, but as we can read in docs:

Note: The spec of a static Pod cannot refer to other API objects (e.g., ServiceAccount, ConfigMap, Secret, etc).

I see that SA is only needed to fetch the ConfigMap from K8s API. I think that you can use --keystone-policy-file instead of --policy-configmap-name and just feed this file directly to the pod as hostPath? Same should be done with certs.

@dulek Ok that sounds promising. I'll give that a try. Thank you!

[heytrav]

I've encountered the same issue running the k8s-keystone-auth application as a static pod.

The static pod starts, but kubeadm init still fails with the same error message.

kubeadm.sh[1666]: error execution phase upload-config/kubeadm: could not bootstrap the admin user in file admin.conf: unable to create ClusterRoleBinding: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline

The pod logs are full of:

Failed to make webhook authorizer request: Post "https://127.0.0.1:8443/webhook?timeout=30s": dial tcp 127.0.0.1:8443: connect: connection refused

It appears that even though the pod runs, it is not actually listening for connections on the host.

Without a serviceAccount in the static pod, k8s-keystone-auth requires access to a kubeconfig in order to run. At init time the options are to

• mount one of the existing kubconfig files
  • admin.conf
  • super-admin.conf
• use kubeadm kubeconfig user --client-name=whatever > /etc/kubernetes/keystone-auth/kubeconfig to generate one

Using admin.conf doesn't work because kubeadm needs to use credentials for the admin user in order to bootstrap the admin user. Generating a custom kubeconfig doesn't work for the same reason. The only conf that works is super-admin.conf which is not ideal.

So far the only solution that has worked for me is to create the cluster without Webhook or the path to the webhook config.

Create the following kustomize definition:

  files:
    - path: /etc/kubernetes/keystone-kustomization/kustomization.yml
      permissions: "0644"
      owner: root:root
      content: |
        resources:
        - kube-apiserver.yaml
        patches:
        - patch: |-
            - op: add
              path: /spec/containers/0/command/-
              value: --authentication-token-webhook-config-file=/etc/kubernetes/webhooks/keystone_webhook_config.yaml
            - op: add
              path: /spec/containers/0/command/-
              value: --authorization-webhook-config-file=/etc/kubernetes/webhooks/keystone_webhook_config.yaml
            - op: add
              path: /spec/containers/0/command/-
              value: --authorization-mode=Webhook
          target:
            kind: Pod

Add the following pre and postKubeadmCommands to modify kube-apiserver.yaml after kubeadm init has completed:

  preKubeadmCommands:
    - mkdir /etc/kubernetes/keystone-kustomization
    - 
  postKubeadmCommands:
    - cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/keystone-kustomization/kube-apiserver.yaml
    - kubectl kustomize /etc/kubernetes/keystone-kustomization -o /etc/kubernetes/manifests/kube-apiserver.yaml

With this setup kubeadm init completes bootstrap of the admin user. After kubeadm init kustomize then modifies the kube-apiserver to use the webhook.

[mnaser]

I think the issue with this is that when/if the cluster is upgraded, the configuration will get wiped.. no?

[heytrav]

The pre/postKubeadmCommands run for an upgrade the same as they do for cluster initialization so kustomize modifies the kube-apiserver.yaml manifest to add the auth webhook.