search

kelseyhightower / kubernetes-the-hard-way

Bootstrap Kubernetes the hard way. No scripts.
Apache License 2.0
40.57k stars 13.92k forks source link

kublet unable to get CSI node related info from kube-apiserver #613

Open trijit08 opened 3 years ago

trijit08 commented 3 years ago

The following error is logged in kube-apiserver audit log

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"5c5321d9-546c-4310-9a93-525611da617d","stage":"ResponseComplete","requestURI":"/apis/storage.k8s.io/v1/csinodes/node1","verb":"get","user":{"username":"system:node:node1","groups":["Kube","system:authenticated"]},"sourceIPs":["35.194.42.18"],"userAgent":"kubelet/v1.18.6 (linux/amd64) kubernetes/dff82dc","objectRef":{"resource":"csinodes","name":"node1","apiGroup":"storage.k8s.io","apiVersion":"v1"},"responseStatus":{"metadata":{},"status":"Failure","reason":"Forbidden","code":403},"requestReceivedTimestamp":"2020-10-31T17:32:55.304278Z","stageTimestamp":"2020-10-31T17:32:55.304507Z","annotations":{"authorization.k8s.io/decision":"forbid","authorization.k8s.io/reason":""}}

And getting below error in kublet log Oct 31 18:18:43 node1 kubelet[32268]: E1031 18:18:43.972203 32268 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:node:node1" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.048924 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.149131 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: I1031 18:18:44.208186 32268 reflector.go:211] Listing and watching v1.Node from k8s.io/kubernetes/pkg/kubelet/kubelet.go:526 Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.210177 32268 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:526: Failed to list v1.Node: nodes "node1" is forbidden: User "system:node:node1" cannot list resource "nodes" in API group "" at the cluster scope Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.249355 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.349512 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.449709 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: I1031 18:18:44.478966 32268 kubelet.go:1993] SyncLoop (housekeeping, skipped): sources aren't ready yet. Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.549897 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: I1031 18:18:44.576380 32268 reflector.go:211] Listing and watching v1.Pod from k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46 Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.578499 32268 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list v1.Pod: pods is forbidden: User "system:node:node1" cannot list resource "pods" in API group "" at the cluster scope Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.650095 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.750268 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.850479 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:44 node1 kubelet[32268]: E1031 18:18:44.950689 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.050836 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.151018 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.251241 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.351448 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.451622 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.551805 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: I1031 18:18:45.621015 32268 reflector.go:211] Listing and watching v1beta1.RuntimeClass from k8s.io/client-go/informers/factory.go:135 Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.622983 32268 reflector.go:178] k8s.io/client-go/informers/factory.go:135: Failed to list v1beta1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:node:node1" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.651997 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.752186 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.852366 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:45 node1 kubelet[32268]: E1031 18:18:45.952572 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.031773 32268 controller.go:136] failed to ensure node lease exists, will retry in 7s, error: leases.coordination.k8s.io "node1" is forbidden: User "system:node:node1" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "kube-node-lease" Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.052823 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.153007 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.253189 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.353411 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.453636 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: I1031 18:18:46.478880 32268 kubelet.go:1993] SyncLoop (housekeeping, skipped): sources aren't ready yet. Oct 31 18:18:46 node1 kubelet[32268]: I1031 18:18:46.496517 32268 eviction_manager.go:243] eviction manager: synchronize housekeeping Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.496587 32268 eviction_manager.go:260] eviction manager: failed to get summary stats: failed to get node info: node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: I1031 18:18:46.498900 32268 kubelet.go:2185] Container runtime status: Runtime Conditions: RuntimeReady=true reason: message:, NetworkReady=true reason: message: Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.553769 32268 kubelet.go:2268] node "node1" not found Oct 31 18:18:46 node1 kubelet[32268]: I1031 18:18:46.565439 32268 reflector.go:211] Listing and watching v1.Service from k8s.io/kubernetes/pkg/kubelet/kubelet.go:517 Oct 31 18:18:46 node1 kubelet[32268]: E1031 18:18:46.567221 32268 reflector.go:178] k8s.io/kubernetes/pkg/kubelet/kubelet.go:517: Failed to list *v1.Service: services is forbidden: User "system:node:node1" cannot list

Please suggest

kbudde commented 3 years ago

Hi,

I had same error messages. Maybe my solution can help you. The hostname of my node was different than the name I used for certificate (system:node:node1 in your case).

See: https://kubernetes.io/docs/reference/access-authn-authz/node/#overview

The value of nodeName must match precisely the name of the node as registered by the kubelet. By default, this is the host name as provided by hostname, or overridden via the kubelet option --hostname-override. However, when using the --cloud-provider kubelet option, the specific hostname may be determined by the cloud provider, ignoring the local hostname and the --hostname-override option. For specifics about how the kubelet determines the hostname, see the kubelet options reference.

My solution was to set the "--hostname-override" option for kubelet to the same name as on the certificate (node1).

fritzduchardt commented 11 months ago

Nice to see you here @kbudde