install openshift 3.10 Cound not find csr for nodes

[wongkafai]

i install openshift 3.10 ,i have always this message: Cound not find csr for nodes: node.XXXX.com. What is the solution?

masters

hostnamectl set-hostname master.XXXX.COM

nodes

hostnamectl set-hostname node.XXXX.COM

masters /etc/hosts And have used DNS

172.31.18.128 master.XXXX.com 172.31.18.129 node.XXXX.com

nodes /etc/hosts And have used DNS

172.31.18.128 master.XXXX.com 172.31.18.129 node.XXXX.com

Inventory file: [OSEv3:children] masters nodes etcd

[OSEv3:vars] ansible_ssh_user=root openshift_deployment_type=origin

[masters] master.XXXX.com

[nodes] master.XXXX.com openshift_node_group_name="node-config-master" openshift_schedulable=True node.XXXX.com openshift_node_group_name="node-config-compute"

[etcd] master.XXXX.com

error message

fatal: [master.XXXX.com]: FAILED! => {"all_subjects_found": [ "subject=/O=system:nodes/CN=system:node:master.XXXX.com\n", "subject=/O=system:nodes/CN=system:node:master.XXXX.com\n", "subject=/O=system:nodes/CN=system:node:master.XXXX.com\n", "subject=/O=system:nodes/CN=system:node:master.XXXX.com\n"], "attempts": 30, "changed": false, "client_approve_results": [], "client_csrs": {}, "failed": true,

"msg": "Could not find csr for nodes: node.XXXX.com", "oc_get_nodes": {"apiVersion": "v1", "items": [{"apiVersion": "v1", "kind": "Node", "metadata": {"annotations": {"volumes.kubernetes.io/controller-managed-attach-detach": "true"}, "creationTimestamp": "2018-11-01T08:59:36Z", "labels": {"beta.kubernetes.io/arch": "amd64", "beta.kubernetes.io/os": "linux", "kubernetes.io/hostname": "master.XXXX.com", "node-role.kubernetes.io/master": "true"}, "name": "master.XXXX.com", "namespace": "", "resourceVersion": "1692", "selfLink": "/api/v1/nodes/master.XXXX.com", "uid": "7517cd70-ddb4-11e8-9a47-00505688190b"}, "spec": {"externalID": "master.XXXX.com"}, "status": {"addresses": [{"address": "172.31.18.128", "type": "InternalIP"}, {"address": "master.XXXX.com", "type": "Hostname"}], "allocatable": {"cpu": "4", "hugepages-1Gi": "0", "hugepages-2Mi": "0", "memory": "32678376Ki", "pods": "250"}, "capacity": {"cpu": "4", "hugepages-1Gi": "0", "hugepages-2Mi": "0", "memory": "32780776Ki", "pods": "250"}, "conditions": [{"lastHeartbeatTime": "2018-11-01T09:05:50Z", "lastTransitionTime": "2018-11-01T08:59:36Z",

"message": "kubelet has sufficient disk space available", "reason": "KubeletHasSufficientDisk", "status": "False", "type": "OutOfDisk"}, {"lastHeartbeatTime": "2018-11-01T09:05:50Z", "lastTransitionTime": "2018-11-01T08:59:36Z",

"message": "kubelet has sufficient memory available", "reason": "KubeletHasSufficientMemory", "status": "False", "type": "MemoryPressure"}, {"lastHeartbeatTime": "2018-11-01T09:05:50Z", "lastTransitionTime": "2018-11-01T08:59:36Z",

"message": "kubelet has no disk pressure", "reason": "KubeletHasNoDiskPressure", "status": "False", "type": "DiskPressure"}, {"lastHeartbeatTime": "2018-11-01T09:05:50Z", "lastTransitionTime": "2018-11-01T08:59:36Z",

"message": "kubelet has sufficient PID available", "reason": "KubeletHasSufficientPID", "status": "False", "type": "PIDPressure"}, {"lastHeartbeatTime": "2018-11-01T09:05:50Z", "lastTransitionTime": "2018-11-01T09:03:10Z",

"message": "kubelet is posting ready status", "reason": "KubeletReady", "status": "True", "type": "Ready"}], "daemonEndpoints": {"kubeletEndpoint": {"Port": 10250}}, "images": [{"names": ["docker.io/openshift/origin-node@sha256:", "docker.io/openshift/origin-node:v3.10"], "sizeBytes": 1272260851}, {"names": ["docker.io/openshift/origin-control-plane@sha256:488fe5f2b65fe4c06ddcd92b0fb8f5d19fdd3ae062ae9e16906d921c5196040f", "docker.io/openshift/origin-control-plane:v3.10"], "sizeBytes": 818655742}, {"names": ["docker.io/openshift/origin-pod@sha256:15170c12dc44a46150fea4fa10819ba5248377a6282e84bee9d320e0fbd6763b", "docker.io/openshift/origin-pod:v3.10.0"], "sizeBytes": 223970687}, {"names": ["quay.io/coreos/etcd@sha256:43fbc8a457aa0cb887da63d74a48659e13947cb74b96a53ba8f47abb6172a948", "quay.io/coreos/etcd:v3.2.22"], "sizeBytes": 37269372}], "nodeInfo": {"architecture": "amd64", "bootID": "1d921d44-6bfd-4ee9-9e29-113b471e5f5e", "containerRuntimeVersion": "docker://1.13.1", "kernelVersion": "3.10.0-957.el7.x86_64", "kubeProxyVersion": "v1.10.0+b81c8f8", "kubeletVersion": "v1.10.0+b81c8f8", "machineID": "3f6d0ce368a544139a31533f90eab800", "operatingSystem": "linux", "osImage": "OpenShift Enterprise", "systemUUID": "0C770842-F6A1-C085-8D99-773B8659D590"}}}], "kind": "List", "metadata": {"resourceVersion": "", "selfLink": ""}}, "rc": 0, "server_approve_results": [], "server_csrs": null, "state": "unknown", "unwanted_csrs": [{"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2018-11-01T09:03:00Z", "generateName": "csr-", "name": "csr-pbmr7", "namespace": "", "resourceVersion": "1449", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-pbmr7", "uid": "ee9ea000-ddb4-11e8-9a47-00505688190b"}, "spec": {"groups": ["system:nodes", "system:authenticated"], "request": "=", "usages": ["digital signature", "key encipherment", "server auth"], "username": "system:node:master.XXXX.com"}, "status": {}}, {"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2018-11-01T08:59:48Z", "generateName": "csr-", "name": "csr-rvkhd", "namespace": "", "resourceVersion": "885", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-rvkhd", "uid": "7c6bd65d-ddb4-11e8-9a47-00505688190b"}, "spec": {"groups": ["system:masters", "system:cluster-admins", "system:authenticated"], "request": "=", "usages": ["digital signature", "key encipherment", "server auth"], "username": "system:admin"}, "status": {}}, {"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2018-11-01T08:59:48Z", "generateName": "csr-", "name": "csr-s77xf", "namespace": "", "resourceVersion": "875", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-s77xf", "uid": "7c3701ef-ddb4-11e8-9a47-00505688190b"}, "spec": {"groups": ["system:masters", "system:cluster-admins", "system:authenticated"], "request": "=", "usages": ["digital signature", "key encipherment", "client auth"], "username": "system:admin"}, "status": {"certificate": "==", "conditions": [{"lastUpdateTime": "2018-11-01T08:59:48Z",

"message": "Auto approving kubelet client certificate after SubjectAccessReview.", "reason": "AutoApproved", "type": "Approved"}]}}, {"apiVersion": "certificates.k8s.io/v1beta1", "kind": "CertificateSigningRequest", "metadata": {"creationTimestamp": "2018-11-01T09:01:56Z", "generateName": "csr-", "name": "csr-wxk5z", "namespace": "", "resourceVersion": "1102", "selfLink": "/apis/certificates.k8s.io/v1beta1/certificatesigningrequests/csr-wxk5z", "uid": "c8711d5e-ddb4-11e8-9a47-00505688190b"}, "spec": {"groups": ["system:nodes", "system:authenticated"], "request": "", "usages": ["digital signature", "key encipherment", "server auth"], "username": "system:node:master.XXXX.com"}, "status": {}}]}

[vrutkovs]

/etc/hosts updates are not taken into account by the kubelet. You'd need to make sure hostname output matches hostname -f output on the host and restart the install.

If you're upgrading you may set openshift_kubelet_name_override option

[michaelgugino]

@wongkafai In this particular instance, it looks like the kubelet service on the node never came up. There is no client-side CSR for the node, either the kubelet never started or there is a network problem between node and master (such as iptables, proxy, etc).