Error : Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, invalid bearer token]]

[gopukrishnantec]

Topic : 10-tls-bootstrapping-kubernetes-workers I have configured the worker-2 with tls bootstrapping but not able to see the csr request in the master. Error getting from master node: Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, invalid bearer token]]

vagrant@master-1:~$ kubectl get nodes --kubeconfig admin.kubeconfig
NAME       STATUS     ROLES    AGE    VERSION
worker-1   NotReady   <none>   137m   v1.13.0

vagrant@master-1:~$ kubectl get csr
No resources found.

In worker-2, bootstrapping config file is mentioned.

vagrant@worker-2:~ ps -aux | grep -i kubelet
root      8022  0.2 14.5 746412 54888 ?        Ssl  08:57   0:00 /usr/local/bin/kubelet --bootstrap-kubeconfig=/var/lib/kubelet/bootstrap-kubeconfig --config=/var/lib/kubelet/kubelet-config.yaml --image-pull-progress-deadline=2m --kubeconfig=/var/lib/kubelet/kubeconfig --cert-dir=/var/lib/kubelet/pki/ --rotate-certificates=true --rotate-server-certificates=true --network-plugin=cni --register-node=true --v=2
vagrant   8091  0.0  0.2  14856  1004 pts/0    S+   08:58   0:00 grep --color=auto -i kubelet

I can see the option is enabled in master :

vagrant@master-1:~ ps -aux | grep -i 'enable-bootstrap-token-auth'
root       939  2.8 15.8 514316 235460 ?       Ssl  08:37   0:43 /usr/local/bin/kube-apiserver --advertise-address=192.168.5.11 --allow-privileged=true --apiserver-count=3 --audit-log-maxage=30 --audit-log-maxbackup=3 --audit-log-maxsize=100 --audit-log-path=/var/log/audit.log --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --client-ca-file=/var/lib/kubernetes/ca.crt --enable-admission-plugins=NodeRestriction,ServiceAccount --enable-swagger-ui=true --enable-bootstrap-token-auth=true --etcd-cafile=/var/lib/kubernetes/ca.crt --etcd-certfile=/var/lib/kubernetes/etcd-server.crt --etcd-keyfile=/var/lib/kubernetes/etcd-server.key --etcd-servers=https://192.168.5.11:2379,https://192.168.5.12:2379 --event-ttl=1h --encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml --kubelet-certificate-authority=/var/lib/kubernetes/ca.crt --kubelet-client-certificate=/var/lib/kubernetes/kube-apiserver.crt --kubelet-client-key=/var/lib/kubernetes/kube-apiserver.key --kubelet-https=true --runtime-config=api/all --service-account-key-file=/var/lib/kubernetes/service-account.crt --service-cluster-ip-range=10.96.0.0/24 --service-node-port-range=30000-32767 --tls-cert-file=/var/lib/kubernetes/kube-apiserver.crt --tls-private-key-file=/var/lib/kubernetes/kube-apiserver.key --v=2

vagrant@worker-2:~ ls -l /var/lib/kubelet/pki/
total 4
-rw------- 1 root root 227 Apr 24 08:21 kubelet-client.key.tmp
vagrant@worker-2:~$ ls /var/lib/kubelet/kubeconfig
ls: cannot access '/var/lib/kubelet/kubeconfig': No such file or directory

I can see below error in /var/log/syslog :

Apr 24 10:00:29 worker-2 kubelet[13253]: I0424 10:00:29.463515   13253 server.go:523] No cloud provider specified: "" from the config file: ""
Apr 24 10:00:29 worker-2 kubelet[13253]: I0424 10:00:29.463532   13253 bootstrap.go:65] Using bootstrap kubeconfig to generate TLS client cert, key and kubeconfig file
Apr 24 10:00:29 worker-2 kubelet[13253]: I0424 10:00:29.469602   13253 bootstrap.go:96] No valid private key and/or certificate found, reusing existing private key or creating a new one
Apr 24 10:00:29 worker-2 kubelet[13253]: I0424 10:00:29.555411   13253 bootstrap.go:239] Failed to connect to apiserver: the server has asked for the client to provide credentials

vagrant@worker-2:~$ cat /var/lib/kubelet/bootstrap-kubeconfig 
cat: /var/lib/kubelet/bootstrap-kubeconfig: Permission denied
vagrant@worker-2:~$ sudo cat /var/lib/kubelet/bootstrap-kubeconfig 
apiVersion: v1
clusters:
- cluster:
    certificate-authority: /var/lib/kubernetes/ca.crt
    server: https://192.168.5.30:6443
  name: bootstrap
contexts:
- context:
    cluster: bootstrap
    user: kubelet-bootstrap
  name: bootstrap
current-context: bootstrap
kind: Config
preferences: {}
users:
- name: kubelet-bootstrap
  user:
    token: 07401b.f395accd246ae52d

From master logs,

Apr 24 10:13:35 master-1 kube-apiserver[939]: E0424 10:13:35.329491     939 authentication.go:65] Unable to authenticate the request due to an error: [invalid bearer token, [invalid bearer token, invalid bearer token]]
authentication from master :

vagrant@master-1:~$ grep -E 'f395accd246ae52d|07401b' bootstrap-token-07401b.yaml 
  name: bootstrap-token-07401b
  token-id: 07401b
  token-secret: f395accd246ae52d
From worker,
vagrant@worker-2:~$ sudo grep -i token /var/lib/kubelet/bootstrap-kubeconfig
    token: 07401b.f395accd246ae52d

What else I could provide to troubleshoot this ?

[willcoderwang]

This is a problem caused by an expired token, as is pointed out in #229 . Just set a valid expiration, and run kubectl apply -f bootstrap-token-07401b.yaml

[fireflycons]

Hi @gopukrishnantec

Please see the recently added (by me) closure comment on #229 which explains this.