search

crc-org / osp4

Proof of Concept repository of CodeReady Containers [Archived]
Apache License 2.0
26 stars 16 forks source link

API server fails to start (0.9.1) #26

Open cmoulliard opened 5 years ago

cmoulliard commented 5 years ago

I have been able on my macbook machine to create the 2 virtualbox vms using ./minishift_virtualbox.sh create && ./minishift_virtualbox.sh start but when I try to log on to the API server, it fails

oc login https://192.168.99.100:6443 -u system:admin
error: dial tcp 192.168.99.100:6443: connect: connection refused - verify you have provided the correct host and port and that the server is currently running.

If I ssh to the master vm and checks the kubelet service, then such connection erros occur

ssh core@127.0.0.1 -p 5222
core@127.0.0.1's password:
Last login: Thu Jan 24 15:38:22 2019 from gateway
Red Hat CoreOS 4.0
 Information: https://url.corp.redhat.com/redhat-coreos
 Bugs: https://github.com/openshift/os

---
[core@test1-master-0 ~]$ systemctl status kubelet.service
● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-01-24 16:46:30 UTC; 59min left
  Process: 2877 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
 Main PID: 2886 (hyperkube)
   Memory: 118.0M
   CGroup: /system.slice/kubelet.service
           └─2886 /usr/bin/hyperkube kubelet --config=/etc/kubernetes/kubelet.conf --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig --rotate-certificates --kubeconfig=/var/lib/kubelet/kubeconfig -...

Jan 24 15:51:54 test1-master-0 hyperkube[2886]: E0124 15:51:54.009137    2886 kubelet_node_status.go:103] Unable to register node "test1-master-0" with API server: Post https://test1-api.tt.testing:6443/api/v1/nodes: dial tcp 192.168.99.100:6443: connect: connection refused
...
Jan 24 15:46:46 test1-master-0 hyperkube[2886]: E0124 15:46:46.477546    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused
Jan 24 15:46:47 test1-master-0 hyperkube[2886]: E0124 15:46:47.489664    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.t...
Jan 24 15:46:47 test1-master-0 hyperkube[2886]: E0124 15:46:47.489672    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused
Jan 24 15:46:47 test1-master-0 hyperkube[2886]: E0124 15:46:47.490960    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://test1-api.tt.testi...
Jan 24 15:46:48 test1-master-0 hyperkube[2886]: E0124 15:46:48.499286    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://test1-api.tt.testi...
Jan 24 15:46:48 test1-master-0 hyperkube[2886]: E0124 15:46:48.499286    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused
Jan 24 15:46:48 test1-master-0 hyperkube[2886]: E0124 15:46:48.499955    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.t...
Jan 24 15:46:49 test1-master-0 hyperkube[2886]: E0124 15:46:49.510005    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.t...
Jan 24 15:46:49 test1-master-0 hyperkube[2886]: E0124 15:46:49.510102    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:464: Failed to list *v1.Node: Get https://test1-api.tt.testi...
Jan 24 15:46:49 test1-master-0 hyperkube[2886]: E0124 15:46:49.511426    2886 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/kubelet.go:455: Failed to list *v1.Service: Get https://te...ction refused

Port opened

[root@test1-master-0 core]# netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      2886/hyperkube
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:46581           0.0.0.0:*               LISTEN      3045/rpc.statd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2936/sshd
tcp        0      0 10.0.3.15:10010         0.0.0.0:*               LISTEN      3040/crio
tcp6       0      0 :::10250                :::*                    LISTEN      2886/hyperkube
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::22                   :::*                    LISTEN      2936/sshd
tcp6       0      0 :::59395                :::*                    LISTEN      3045/rpc.statd
gbraad commented 5 years ago

The port is not opened and it clearly shows issues with the API server. Have you restarted?

cmoulliard commented 5 years ago

If I do ./minishift_virtualbox.sh stop followed by ./minishift_virtualbox.sh start, then the same error is reported

ssh core@192.168.99.100
core@192.168.99.100's password:
Last login: Thu Jan 24 15:57:06 2019 from 192.168.99.1
Red Hat CoreOS 4.0
 Information: https://url.corp.redhat.com/redhat-coreos
 Bugs: https://github.com/openshift/os

---
[core@test1-master-0 ~]$ sudo lsof -i -P -n | grep LISTEN
sudo: lsof: command not found
[core@test1-master-0 ~]$ sudo su
[root@test1-master-0 core]# netstat -tulpn | grep LISTEN
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      2899/hyperkube
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 0.0.0.0:39990           0.0.0.0:*               LISTEN      3033/rpc.statd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2938/sshd
tcp        0      0 10.0.3.15:10010         0.0.0.0:*               LISTEN      3030/crio
tcp6       0      0 :::10250                :::*                    LISTEN      2899/hyperkube
tcp6       0      0 :::60906                :::*                    LISTEN      3033/rpc.statd
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::22                   :::*                    LISTEN      2938/sshd

and

Jan 24 16:11:51 test1-master-0 hyperkube[2899]: E0124 16:11:51.892882    2899 reflector.go:136] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:47: Failed to list *v1.Pod: Get https://test1-api.tt.testing:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dtest1-master-0&limit=500&resourceVersion=0: dial tcp 192.168.99.100:6443: connect: connection refused

Why kubelet is not able to access the API server ?

cmoulliard commented 5 years ago

Here is what crictl reports 

[root@test1-master-0 core]# crictl images
IMAGE                                                 TAG                                        IMAGE ID            SIZE
k8s.gcr.io/pause                                      3.1                                        da86e6ba6ca19       747kB
quay.io/coreos/etcd                                   v3.3.10                                    643c21638c1c9       39.7MB
quay.io/coreos/kube-client-agent                      36c62ccd7b16b522450c61e96fc556b217ee24f5   b02de22ff740f       33.8MB
quay.io/openshift-release-dev/ocp-release             <none>                                     d0bc83da1db0c       292MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     1484c1de242d8       316MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     eeee805d24429       576MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     007942e83b562       288MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     e0972dab0d058       285MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     430af544425fa       374MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     73f1f98cd6dd9       316MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     169a72c94dcde       281MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     9d15e9bac6339       366MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     c779d02a634cc       315MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     e243d7975e99c       285MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     bfa0ad6d2753f       295MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     d56d47ed50c2f       374MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     e7ebbc9e5eb03       514MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     edca0ee739703       314MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     0ee50df593f6d       312MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     7f8e4a1588a21       268MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     1c9c9fa847ef1       293MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     66499234b190f       299MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     b0d2d081a1eeb       287MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     090cff599214a       288MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     0b374c5262383       294MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     40583808760aa       317MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     1319a581abe9a       311MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     2e30054dab0ab       322MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     f5a02bf81b079       290MB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     459d56445a917       1.25GB
quay.io/openshift-release-dev/ocp-v4.0                <none>                                     b867876bda666       278MB
registry.svc.ci.openshift.org/openshift/origin-v4.0   setup-etcd-environment                     ef87ffbfa30dc       248MB
[root@test1-master-0 core]# crictl pods
POD ID              CREATED             STATE               NAME                                                              NAMESPACE                                                 ATTEMPT
dd180d0bbbf4a       12 minutes ago      NotReady            etcd-member-test1-master-0                                        kube-system                                               1
27c1ec3ad57cf       12 minutes ago      NotReady            openshift-kube-apiserver-test1-master-0                           openshift-kube-apiserver                                  1
eb2eb97c751a3       12 minutes ago      NotReady            openshift-kube-controller-manager-test1-master-0                  openshift-kube-controller-manager                         1
7981a2cc5f19f       12 minutes ago      NotReady            openshift-kube-scheduler-test1-master-0                           openshift-kube-scheduler                                  1
2bfcea9198333       About an hour ago   NotReady            cluster-autoscaler-operator-6855f55d94-29jkx                      openshift-cluster-api                                     1
9bdcfa7ed3115       About an hour ago   NotReady            dns-default-9kcsf                                                 openshift-dns                                             1
c96bde52fc5e9       About an hour ago   NotReady            machine-config-operator-769967ddf5-sspjj                          openshift-machine-config-operator                         1
fd0e773926d5e       About an hour ago   NotReady            origin-cluster-osin-operator-774dc44fd8-rs76w                     openshift-core-operators                                  1
a879cd9573fae       About an hour ago   NotReady            openshift-console-85d4d65d64-mbgpk                                openshift-console                                         1
7bcee689097c1       About an hour ago   NotReady            openshift-service-cert-signer-operator-784d9677b5-vlr2m           openshift-core-operators                                  1
b53a20cece153       About an hour ago   NotReady            machine-api-operator-67f964b4d-tdnk5                              openshift-cluster-api                                     1
fe43242899947       About an hour ago   NotReady            configmap-cabundle-injector-755f877875-wv886                      openshift-service-cert-signer                             1
6e0787fe49312       About an hour ago   NotReady            openshift-cluster-openshift-controller-manager-operator-5dwhnht   openshift-cluster-openshift-controller-manager-operator   1
7a7b18a366ab1       About an hour ago   NotReady            openshift-kube-controller-manager-operator-66cbb459f8-6n6dg       openshift-kube-controller-manager-operator                1
7c60c5fa3881c       About an hour ago   NotReady            origin-cluster-osin-operator2-669747b677-txl8z                    openshift-core-operators                                  1
c6121c5b5ddbe       About an hour ago   NotReady            dns-operator-6dbdd7df84-lvm86                                     openshift-dns-operator                                    1
fa7bfba3ee485       About an hour ago   NotReady            openshift-console-85d4d65d64-qbppk                                openshift-console                                         1
b48256d4085c7       About an hour ago   NotReady            openshift-cluster-kube-scheduler-operator-7776d4f85c-2s7t6        openshift-cluster-kube-scheduler-operator                 1
91b89b8555075       About an hour ago   NotReady            image-registry-6fb8cbfb4f-q4sqt                                   openshift-image-registry                                  1
5a6da8e3d57c7       About an hour ago   NotReady            openshift-console-85d4d65d64-k8bbt                                openshift-console                                         1
afcebd9899048       About an hour ago   NotReady            cluster-samples-operator-7cf8d8cf66-svddq                         openshift-cluster-samples-operator                        1
6c73717a1259f       About an hour ago   NotReady            apiservice-cabundle-injector-7cc897d85b-pbwh5                     openshift-service-cert-signer                             1
a1af578009df6       About an hour ago   NotReady            clusterapi-manager-controllers-748cb8d66c-68c8w                   openshift-cluster-api                                     1
ef4f10b5a8586       About an hour ago   NotReady            registry-ca-hostmapper-w6l68                                      openshift-image-registry                                  1
4bc7fd9f6280d       About an hour ago   NotReady            openshift-kube-apiserver-operator-58b8c455c5-4jhqg                openshift-kube-apiserver-operator                         1
dc63b9ad5b246       About an hour ago   NotReady            apiserver-sf68z                                                   openshift-apiserver                                       1
1a5a3d2320d77       About an hour ago   NotReady            machine-config-controller-68f5989588-p45v9                        openshift-machine-config-operator                         1
77dc3704f4e05       About an hour ago   NotReady            service-serving-cert-signer-66f9694c4d-8l878                      openshift-service-cert-signer                             1
8c574c82313ce       About an hour ago   NotReady            openshift-apiserver-operator-77756dfc66-942df                     openshift-apiserver-operator                              1
d6d32c2801713       About an hour ago   NotReady            controller-manager-6qp2c                                          openshift-controller-manager                              1
d265f3412b6c4       About an hour ago   NotReady            cluster-image-registry-operator-6c4478c54c-kb7rh                  openshift-image-registry                                  1
965abe7e454a9       About an hour ago   NotReady            tuned-z77hj                                                       openshift-cluster-node-tuning-operator                    1
2b6590f633081       About an hour ago   NotReady            machine-config-server-jp594                                       openshift-machine-config-operator                         1
4c9eead87f8a8       About an hour ago   NotReady            machine-config-daemon-s4xxc                                       openshift-machine-config-operator                         1
937d3ebb7ecf9       About an hour ago   NotReady            machine-approver-86b68b66f7-wvztd                                 openshift-cluster-machine-approver                        1
d02712f26ecd2       About an hour ago   NotReady            cluster-network-operator-vx7xr                                    openshift-cluster-network-operator                        1
297926230d315       About an hour ago   NotReady            sdn-nzdgx                                                         openshift-sdn                                             1
9868e0dede288       About an hour ago   NotReady            sdn-controller-6svcg                                              openshift-sdn                                             1
aa0860f318bb7       About an hour ago   NotReady            cluster-version-operator-7b47d58bff-j6nb7                         openshift-cluster-version                                 1
944f8b6783bc8       About an hour ago   NotReady            ovs-xfdg6                                                         openshift-sdn                                             1
4bb64351080a3       2 weeks ago         NotReady            installer-3-test1-master-0                                        openshift-kube-apiserver                                  0
3b1c8e1f2d4f6       2 weeks ago         NotReady            installer-1-test1-master-0                                        openshift-kube-controller-manager                         0
20bcea5fc476e       2 weeks ago         NotReady            installer-2-test1-master-0                                        openshift-kube-apiserver                                  0
791d3200beeb8       2 weeks ago         NotReady            installer-1-test1-master-0                                        openshift-kube-apiserver                                  0
28629cdfda637       2 weeks ago         NotReady            installer-1-test1-master-0                                        openshift-kube-scheduler                                  0
[root@test1-master-0 core]#

and

[root@test1-master-0 core]# crictl inspectp 27c1ec3ad57cf
{
  "status": {
    "id": "27c1ec3ad57cfcb9d101acebbc53314d17d105bdeafbf6fa5715be4ffdf1dd84",
    "metadata": {
      "attempt": 1,
      "name": "openshift-kube-apiserver-test1-master-0",
      "namespace": "openshift-kube-apiserver",
      "uid": "8f63e2613f94443f7263843d3501805f"
    },
    "state": "SANDBOX_NOTREADY",
    "createdAt": "2019-01-24T16:01:25.232664761Z",
    "network": {
      "ip": ""
    },
    "linux": {
      "namespaces": {
        "options": {
          "ipc": "POD",
          "network": "NODE",
          "pid": "CONTAINER"
        }
      }
    },
    "labels": {
      "apiserver": "true",
      "app": "openshift-kube-apiserver",
      "io.kubernetes.container.name": "POD",
      "io.kubernetes.pod.name": "openshift-kube-apiserver-test1-master-0",
      "io.kubernetes.pod.namespace": "openshift-kube-apiserver",
      "io.kubernetes.pod.uid": "8f63e2613f94443f7263843d3501805f",
      "revision": "3"
    },
    "annotations": {
      "kubernetes.io/config.hash": "8f63e2613f94443f7263843d3501805f",
      "kubernetes.io/config.seen": "2019-01-24T16:01:24.098343869Z",
      "kubernetes.io/config.source": "file"
    }
  },
  "version": {
    "version": "1.11.10"
  }
}
[root@test1-master-0 core]# crictl logs 27c1ec3ad57cf
E0124 16:15:35.331505    3977 remote_runtime.go:278] ContainerStatus "27c1ec3ad57cf" from runtime service failed: rpc error: code = Unknown desc = specified container not found: 27c1ec3ad57cfcb9d101acebbc53314d17d105bdeafbf6fa5715be4ffdf1dd84
FATA[0000] rpc error: code = Unknown desc = specified container not found: 27c1ec3ad57cfcb9d101acebbc53314d17d105bdeafbf6fa5715be4ffdf1dd84
[root@test1-master-0 core]#
gbraad commented 5 years ago

27c1ec3ad57cf 12 minutes ago NotReady openshift-kube-apiserver-test1-master-0

Not working as expected. Let's disuss this tomorrow, as @praveenkumar and I need to also verify this. It might be related to pruning.

cmoulliard commented 5 years ago

If I try to create a pod then it fails with this error 

[root@test1-master-0 core]# crictl runp pod-config.json
FATA[0000] run pod sandbox failed: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_nginx-sandbox_default_hdishd83djaidwnduwk28bcsb_1(81dbaeafd1d872b88debd40df631d786d2642aff6653a94403b322deda046fa7): OpenShift SDN network process is not (yet?) available
gbraad commented 5 years ago

OpenShift SDN will not start properly due to #17