search

kontena / pharos-cluster

Pharos - The Kubernetes Distribution
https://k8spharos.dev/
Apache License 2.0
311 stars 43 forks source link

Cri-o is broken after reboot #538

Closed jakolehm closed 6 years ago

jakolehm commented 6 years ago 
-- Logs begin at Thu 2018-08-16 09:27:22 UTC, end at Thu 2018-08-16 09:30:26 UTC. --
Aug 16 09:27:25 pharos-worker-0 systemd[1]: Starting Open Container Initiative Daemon...
Aug 16 09:27:25 pharos-worker-0 sysctl[1463]: net.ipv4.ip_forward = 1
Aug 16 09:27:25 pharos-worker-0 crio[1502]: time="2018-08-16 09:27:25.844253805Z" level=info msg="[graphdriver] using prior storage driver: overlay"
Aug 16 09:27:25 pharos-worker-0 crio[1502]: time="2018-08-16 09:27:25.862101545Z" level=info msg="CNI network pharos (type=weave-net) is used from /etc/cni/net.d/00-pharos.conflist"
Aug 16 09:27:25 pharos-worker-0 crio[1502]: time="2018-08-16 09:27:25.862131899Z" level=info msg="Initial CNI setting succeeded"
Aug 16 09:27:25 pharos-worker-0 crio[1502]: time="2018-08-16 09:27:25.938674650Z" level=warning msg="could not restore sandbox b01bd69f5856e81c7d77e276b054535b444cf0531fb9792a87a583c948d213e1 container b01bd69f5856e81c7d77e276b054535b444cf0531fb9792a87a583c948d213e1: open /var/run/containers/storage/overlay-containers/b01bd69f5856e81c7d77e276b054535b444cf0531fb9792a87a583c948d213e1/userdata/config.json: no such file or directory"
Aug 16 09:27:25 pharos-worker-0 crio[1502]: time="2018-08-16 09:27:25.939048911Z" level=warning msg="could not restore sandbox ef57ba8a8665dac778a0e74390469f07d3dedba6bc0e10f32d346f86f831e547 container ef57ba8a8665dac778a0e74390469f07d3dedba6bc0e10f32d346f86f831e547: open /var/run/containers/storage/overlay-containers/ef57ba8a8665dac778a0e74390469f07d3dedba6bc0e10f32d346f86f831e547/userdata/config.json: no such file or directory"
Aug 16 09:27:25 pharos-worker-0 crio[1502]: time="2018-08-16 09:27:25.940122218Z" level=warning msg="could not restore container 36eac66118ac7190b18bb6897b3624414c325002e605f2e3834cce562c5bf2fc: open /var/run/containers/storage/overlay-containers/36eac66118ac7190b18bb6897b3624414c325002e605f2e3834cce562c5bf2fc/userdata/config.json: no such file or directory"
Aug 16 09:27:25 pharos-worker-0 crio[1502]: time="2018-08-16 09:27:25.940200878Z" level=warning msg="could not restore container 5acbcc9628180dd383fbad7cc07ab3e53fdf6e56a0a8c0373916c2742789ad07: open /var/run/containers/storage/overlay-containers/5acbcc9628180dd383fbad7cc07ab3e53fdf6e56a0a8c0373916c2742789ad07/userdata/config.json: no such file or directory"
Aug 16 09:27:25 pharos-worker-0 systemd[1]: Started Open Container Initiative Daemon.
SpComb commented 6 years ago

Associated errors:

Aug 16 07:34:41 pharos-worker-1 kubelet[1716]: E0816 07:34:41.051225    1716 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = pod sandbox with name "k8s_pharos-proxy-pharos-worker-1_kube-system_74e34eee2b5335d5224700ea81fdd69f_0" already exists
Aug 16 07:34:41 pharos-worker-1 kubelet[1716]: E0816 07:34:41.051794    1716 kuberuntime_sandbox.go:56] CreatePodSandbox for pod "pharos-proxy-pharos-worker-1_kube-system(74e34eee2b5335d5224700ea81fdd69f)" failed: rpc error: code = Unknown desc = pod sandbox with name "k8s_pharos-proxy-pharos-worker-1_kube-system_74e34eee2b5335d5224700ea81fdd69f_0" already exists
Aug 16 07:34:41 pharos-worker-1 kubelet[1716]: E0816 07:34:41.052180    1716 kuberuntime_manager.go:646] createPodSandbox for pod "pharos-proxy-pharos-worker-1_kube-system(74e34eee2b5335d5224700ea81fdd69f)" failed: rpc error: code = Unknown desc = pod sandbox with name "k8s_pharos-proxy-pharos-worker-1_kube-system_74e34eee2b5335d5224700ea81fdd69f_0" already exists
Aug 16 07:34:41 pharos-worker-1 kubelet[1716]: E0816 07:34:41.052630    1716 pod_workers.go:186] Error syncing pod 74e34eee2b5335d5224700ea81fdd69f ("pharos-proxy-pharos-worker-1_kube-system(74e34eee2b5335d5224700ea81fdd69f)"), skipping: failed to "CreatePodSandbox" for "pharos-proxy-pharos-worker-1_kube-system(74e34eee2b5335d5224700ea81fdd69f)" with CreatePodSandboxError: "CreatePodSandbox for pod \"pharos-proxy-pharo
Aug 16 07:43:24 pharos-worker-1 crio[23525]: time="2018-08-16 07:43:24.910330178Z" level=warning msg="could not restore sandbox 6ebf512cbe82c12762cd6cc27a9c456e0af726facd36f039457403f1d9af6227 container 6ebf512cbe82c12762cd6cc27a9c456e0af726facd36f039457403f1d9af6227: open /var/run/containers/storage/overlay-containers/6ebf512cbe82c12762cd6cc27a9c456e0af726facd36f039457403f1d9af6227/userdata/config.json: no such file or directory"
jakolehm commented 6 years ago

This happens only with cri-o 1.11.1, older versions work fine.

jnummelin commented 6 years ago

Reported upstream: https://github.com/kubernetes-incubator/cri-o/issues/1742

SpComb commented 6 years ago

Fixed in https://github.com/kubernetes-incubator/cri-o/pull/1744 + https://github.com/kubernetes-incubator/cri-o/pull/1745 pending 1.11.2 release