search

Mirantis / virtlet

Kubernetes CRI implementation for running VM workloads
Apache License 2.0
743 stars 128 forks source link

Does Virtlet support OVN as a CNI plugin? #796

Closed amiribrahim closed 5 years ago

amiribrahim commented 6 years ago

I've setup a local k8s cluster of a master and two nodes. OVN is my CNI plugin. I was able to create ngnix pods. Installed virtlet and this is the output of kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE default apachetwin 1/1 Running 0 17h 192.168.2.3 k8sminion1 default nginxtwin 1/1 Running 0 17h 192.168.3.3 k8sminion2 kube-system coredns-576cbf47c7-b7plk 1/1 Running 4 17h 192.168.1.3 k8smaster kube-system coredns-576cbf47c7-rts69 1/1 Running 4 17h 192.168.1.4 k8smaster kube-system etcd-k8smaster 1/1 Running 0 17h 10.10.0.11 k8smaster kube-system kube-apiserver-k8smaster 1/1 Running 0 17h 10.10.0.11 k8smaster kube-system kube-controller-manager-k8smaster 1/1 Running 0 17h 10.10.0.11 k8smaster kube-system kube-proxy-f28pq 1/1 Running 0 17h 10.10.0.11 k8smaster kube-system kube-proxy-gvbhj 1/1 Running 0 17h 10.10.0.13 k8sminion2 kube-system kube-proxy-pcjff 1/1 Running 0 17h 10.10.0.12 k8sminion1 kube-system kube-scheduler-k8smaster 1/1 Running 0 17h 10.10.0.11 k8smaster kube-system virtlet-j4gwn 3/3 Running 0 17h 10.10.0.13 k8sminion2 kube-system virtlet-kchvf 3/3 Running 0 17h 10.10.0.12 k8sminion1

But each tile I try to create a vm pod using this kubectl create -f https://raw.githubusercontent.com/Mirantis/virtlet/master/examples/cirros-vm.yaml I get : default cirros-vm 0/1 ErrImagePull 0 47s k8sminion1 default cirros-vm 0/1 ImagePullBackOff 0 48s 192.168.2.4 k8sminion1 default cirros-vm 0/1 ErrImagePull 0 2m1s 192.168.2.4 k8sminion1 default cirros-vm 0/1 ImagePullBackOff 0 2m14s 192.168.2.4 k8sminion1 default cirros-vm 0/1 ErrImagePull 0 3m14s 192.168.2.4 k8sminion1 default cirros-vm 0/1 ImagePullBackOff 0 3m29s 192.168.2.4 k8sminion1

I'm not sure weather the reason is the CNI plugin I'm using or other things Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

Server Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:43:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"} I've used latest criproxy and virtlet versions

lukaszo commented 6 years ago

Please attach output from the following commands: kubectl describe pod cirros-vm kubectl -n kube-system logs virtlet-kchvf -c virtlet

jellonek commented 6 years ago

Did you tried a simplier setup, e.g. with flannel plugin? Error message seems to be disconnected with CNI at all and it suggests that there is other issue, with downloading images by virtlet.

amiribrahim commented 6 years ago

vagrant@k8smaster:~$ kubectl describe pod cirros-vm Name: cirros-vm Namespace: default Priority: 0 PriorityClassName: Node: k8sminion2/10.10.0.13 Start Time: Tue, 06 Nov 2018 15:35:54 +0000 Labels: Annotations: VirtletDiskDriver: virtio VirtletRootVolumeSize: 1Gi VirtletSSHKeys: ssh-rsa ... kubernetes.io/target-runtime: virtlet.cloud ovn: {"ip_address":"192.168.3.4/24", "mac_address":"0a:00:00:00:00:06", "gateway_ip": "192.168.3.1"} Status: Pending IP: 192.168.3.4 Containers: cirros-vm: Container ID:
Image: virtlet.cloud/cirros Image ID:
Port: Host Port: State: Waiting Reason: ImagePullBackOff Ready: False Restart Count: 0 Limits: memory: 160Mi Requests: memory: 160Mi Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-gdwmv (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: default-token-gdwmv: Type: Secret (a volume populated by a Secret) SecretName: default-token-gdwmv Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message

Normal Scheduled 64s default-scheduler Successfully assigned default/cirros-vm to k8sminion2 Warning Failed 18s kubelet, k8sminion2 Failed to pull image "virtlet.cloud/cirros": rpc error: code = Unknown desc = Error response from daemon: Get https://virtlet.cloud/v1/_ping: dial tcp 184.168.131.241:443: i/o timeout Warning Failed 18s kubelet, k8sminion2 Error: ErrImagePull Normal BackOff 18s kubelet, k8sminion2 Back-off pulling image "virtlet.cloud/cirros" Warning Failed 18s kubelet, k8sminion2 Error: ImagePullBackOff Normal Pulling 6s (x2 over 63s) kubelet, k8sminion2 pulling image "virtlet.cloud/cirros"

vagrant@k8smaster:~$ kubectl -n kube-system logs virtlet-kchvf -c virtlet Waiting for libvirt... Waiting for libvirt... I1105 22:23:47.232409 11550 listener.go:66] UnixSocket Listener started W1105 22:23:47.276563 11550 image.go:552] StripTags: failed to parse image name as ref: "": repository name must have at least one component I1105 22:23:47.276618 11550 manager.go:151] Starting server on socket /run/virtlet.sock 2018/11/05 22:23:47 transport: http2Server.HandleStreams failed to receive the preface from client: EOF 2018/11/05 22:23:57 transport: http2Server.HandleStreams failed to receive the preface from client: EOF 2018/11/05 22:24:07 transport: http2Server.HandleStreams failed to receive the preface from client: EOF 2018/11/05 22:24:17 transport: http2Server.HandleStreams failed to receive the preface from client: EOF 2018/11/05 22:24:27 transport: http2Server.HandleStreams failed to receive the preface from client: EOF

jellonek commented 6 years ago

How you installed virtlet? Looks like you didn't reconfigured kubelet to use criproxy. Please follow this and this documents, keeping in mind that for criproxy you need to follow also "Installation on a systemd-based system in a general case" section, even if installation was based on kubeadm.

amiribrahim commented 6 years ago

jellonek, I did as advised, but receiving the same output as above. To make sure I'm doing this the right way:

  1. Do I need to install criproxy on master?
  2. How should I do this step: Then we need to reconfigure kubelet. You need to pass the following extra flags to it to make it use CRI Proxy (you will need to do another systemctl daemon-reload):

--container-runtime=remote \ --container-runtime-endpoint=unix:///run/criproxy.sock \ --image-service-endpoint=unix:///run/criproxy.sock \ --enable-controller-attach-detach=false

I've modified the /etc/systemd/system/dockershim.service file with the extra flags and then systemctl daemon-reload

jellonek commented 6 years ago

criproxy is needed on compute (or worker, whatever you call them) nodes. Mentioned flags should be set in /etc/systemd/system/kubelet.service (that's where you have to reconfigure kubelet), not in dockershim.service.

amiribrahim commented 6 years ago

There is no such file /etc/systemd/system/kubelet.service, it is a kubeadm cluster. the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf file contains:

Note: This dropin only works with kubeadm and kubelet v1.11+

[Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"

This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically

EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env

This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use

the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.

EnvironmentFile=-/etc/default/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

lukaszo commented 6 years ago

@amiribrahim Please use code formatting when pasting file content: ``` ```. It will be easier to use: https://help.github.com/articles/basic-writing-and-formatting-syntax/

jellonek commented 6 years ago

Ok. In this case kubelet is started probably from other file - /lib/systemd/system/kubelet.service.

amiribrahim commented 6 years ago

Environment setup details:

lukaszo commented 6 years ago

please show output of ps ax|grep kubelet from both minions

amiribrahim commented 6 years ago
lukaszo commented 6 years ago

As you can see kubelet is still running with old flags. Did you reload systemd and restarted service?

amiribrahim commented 6 years ago

I did. After adding these flags to /lib/systemd/system/kubelet.service:

--container-runtime=remote \
--container-runtime-endpoint=unix:///run/criproxy.sock \
--image-service-endpoint=unix:///run/criproxy.sock \
--enable-controller-attach-detach=false

I restarted kubelet service this way:

sudo systemctl daemon-reload
sudo systemctl restart kubelet
lukaszo commented 6 years ago

What says sudo systemctl status kubelet?

amiribrahim commented 6 years ago
  1. On minion1: 
    
vagrant@k8sminion1:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/kubelet.service.d
       └─10-kubeadm.conf, 99-criproxy.conf
Active: active (running) since Fri 2018-11-09 18:59:48 UTC; 6min ago
 Docs: https://kubernetes.io/docs/home/
Main PID: 21311 (kubelet)
Tasks: 16
Memory: 35.8M
  CPU: 11.417s
CGroup: /system.slice/kubelet.service
       └─21311 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgrou

Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442705 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "k8s-pods-dir" (UniqueNam Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442757 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "libvirt-log" (UniqueName Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442802 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "image-name-translations" Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442843 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueNam Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442877 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "dockersock" (UniqueName: Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442912 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "libvirt" (UniqueName: "k Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442947 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "libvirt-sockets" (Unique Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.442984 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "netns-dir" (UniqueName: Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.443023 21311 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "virtlet-token-ppjfz" (Un Nov 09 18:59:49 k8sminion1 kubelet[21311]: I1109 18:59:49.443046 21311 reconciler.go:154] Reconciler: start to sync state

1. On minion2 (trying to create a VM):

vagrant@k8sminion2:~$ sudo systemctl status kubelet ● kubelet.service - kubelet: The Kubernetes Node Agent Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-kubeadm.conf, 99-criproxy.conf Active: active (running) since Fri 2018-11-09 18:59:35 UTC; 8min ago Docs: https://kubernetes.io/docs/home/ Main PID: 25786 (kubelet) Tasks: 17 Memory: 37.8M CPU: 15.304s CGroup: /system.slice/kubelet.service └─25786 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgrou

Nov 09 19:06:13 k8sminion2 kubelet[25786]: E1109 19:06:13.311536 25786 kuberuntime_manager.go:744] container start failed: ErrImagePull: rpc error: code = Unknown desc = Error respons Nov 09 19:06:13 k8sminion2 kubelet[25786]: E1109 19:06:13.311562 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:06:28 k8sminion2 kubelet[25786]: E1109 19:06:28.301818 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:06:43 k8sminion2 kubelet[25786]: E1109 19:06:43.307803 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:06:55 k8sminion2 kubelet[25786]: E1109 19:06:55.338965 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:07:08 k8sminion2 kubelet[25786]: E1109 19:07:08.306215 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:07:20 k8sminion2 kubelet[25786]: E1109 19:07:20.301176 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:07:31 k8sminion2 kubelet[25786]: E1109 19:07:31.336915 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:07:46 k8sminion2 kubelet[25786]: E1109 19:07:46.302141 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e Nov 09 19:08:00 k8sminion2 kubelet[25786]: E1109 19:08:00.300188 25786 pod_workers.go:186] Error syncing pod b0468999-e392-11e8-9396-02ac2e7f59cd ("cirros-vm_default(b0468999-e392-11e

lukaszo commented 6 years ago

hmm, maybe you added the flags in a wrong place in /lib/systemd/system/kubelet.service Can you show it?

amiribrahim commented 6 years ago 
# /lib/systemd/system/kubelet.service
[Unit]
Description=kubelet: The Kubernetes Node Agent
Documentation=https://kubernetes.io/docs/home/

[Service]
ExecStart=/usr/bin/kubelet --container-runtime=remote \
--container-runtime-endpoint=unix:///run/criproxy.sock \
--image-service-endpoint=unix:///run/criproxy.sock \
--enable-controller-attach-detach=false
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
WantedBy=multi-user.target
amiribrahim commented 6 years ago

For your reference

# /etc/systemd/system/criproxy.service
[Unit]
Description=CRI Proxy

[Service]
ExecStart=/usr/local/bin/criproxy -v 3 -logtostderr -connect /var/run/dockershim.sock,virtlet.cloud:/run/virtlet.sock -listen /run/criproxy.sock
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=kubelet.service

and 

# /etc/systemd/system/dockershim.service
[Unit]
Description=dockershim for criproxy

[Service]
ExecStart=/usr/bin/kubelet --experimental-dockershim --port 11250 --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=cgroupfs --network-plugin=cni
Restart=always
StartLimitInterval=0
RestartSec=10

[Install]
RequiredBy=criproxy.service
lukaszo commented 6 years ago

I think some parts of it are overwritten in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf Can you show it?

Also you can read more about systemd drop-in mechanism here https://coreos.com/os/docs/latest/using-systemd-drop-in-units.html

amiribrahim commented 6 years ago

/etc/systemd/system/kubelet.service.d/10-kubeadm.conf

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
lukaszo commented 6 years ago

Exactly, everything is overwritten here. Modify the last line to look like this:

ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --container-runtime=remote --container-runtime-endpoint=unix:///run/criproxy.sock --image-service-endpoint=unix:///run/criproxy.sock --enable-controller-attach-detach=false

and reload and restart kubelet

amiribrahim commented 6 years ago

I did as advised, added the flags to /etc/systemd/system/kubelet.service.d/10-kubeadm.conf reloaded and restarted kubelet But now on master:

vagrant@k8smaster:~$ kubectl get nodes
NAME         STATUS     ROLES    AGE   VERSION
k8smaster    Ready      master   2d    v1.12.2
k8sminion1   NotReady   <none>   2d    v1.12.2
k8sminion2   NotReady   <none>   2d    v1.12.2
lukaszo commented 6 years ago

check the logs using journalctl -u kubelet

amiribrahim commented 6 years ago

It is a configuration issue as you expected 

Nov 09 20:30:43 k8sminion1 kubelet[11880]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https
Nov 09 20:30:43 k8sminion1 kubelet[11880]: Flag --enable-controller-attach-detach has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
Nov 09 20:30:43 k8sminion1 kubelet[11880]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https
Nov 09 20:30:43 k8sminion1 kubelet[11880]: Flag --enable-controller-attach-detach has been deprecated, This parameter should be set via the config file specified by the Kubelet's --conf
lukaszo commented 6 years ago

Those are only warnings

amiribrahim commented 6 years ago

I believe that reason is in somewhere below, appreciate your help

Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.359534   13222 server.go:408] Version: v1.12.2
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.360053   13222 plugins.go:99] No cloud provider specified.
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.378678   13222 certificate_store.go:131] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.398615   13222 server.go:667] --cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.398905   13222 container_manager_linux.go:247] container manager verified user specified cgroup-root exists: []
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.398924   13222 container_manager_linux.go:252] Creating Container Manager object based on Node Config: {RuntimeCgroupsName: Sys
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.398993   13222 container_manager_linux.go:271] Creating device plugin manager: true
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.399015   13222 state_mem.go:36] [cpumanager] initializing new in-memory state store
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.399091   13222 state_mem.go:84] [cpumanager] updated default cpuset: ""
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.399101   13222 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.399168   13222 kubelet.go:279] Adding pod path: /etc/kubernetes/manifests
Nov 09 21:01:07 k8sminion1 kubelet[13222]: I1109 21:01:07.399188   13222 kubelet.go:304] Watching apiserver
Nov 09 21:03:07 k8sminion1 kubelet[13222]: E1109 21:03:07.403281   13222 remote_runtime.go:72] Version from runtime service failed: rpc error: code = DeadlineExceeded desc = context dea
Nov 09 21:03:07 k8sminion1 kubelet[13222]: E1109 21:03:07.403523   13222 kuberuntime_manager.go:183] Get runtime version failed: rpc error: code = DeadlineExceeded desc = context deadli
Nov 09 21:03:07 k8sminion1 kubelet[13222]: F1109 21:03:07.403587   13222 server.go:262] failed to run Kubelet: failed to create kubelet: rpc error: code = DeadlineExceeded desc = contex
Nov 09 21:03:07 k8sminion1 systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Nov 09 21:03:07 k8sminion1 systemd[1]: kubelet.service: Unit entered failed state.
Nov 09 21:03:07 k8sminion1 systemd[1]: kubelet.service: Failed with result 'exit-code'.
Nov 09 21:03:17 k8sminion1 systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Nov 09 21:03:17 k8sminion1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
Nov 09 21:03:17 k8sminion1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
lukaszo commented 6 years ago

This one is important Nov 09 21:03:07 k8sminion1 kubelet[13222]: E1109 21:03:07.403281 13222 remote_runtime.go:72] Version from runtime service failed: rpc error: code = DeadlineExceeded desc = context dea It looks like kubelet can not connect with criproxy. Is it running? If yeas please share its logs

amiribrahim commented 6 years ago

On both minions, all logs are:

Nov 11 18:05:05 k8sminion2 criproxy[7618]: I1111 18:05:05.757006    7618 utils.go:48] attempt 329789: can't connect to "/var/run/dockershim.sock" yet: dial unix /var/run/dockershim.sock
lukaszo commented 6 years ago

The log clearly says what's wrong. There's issue with the dockershim service

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.