clearcontainers / proxy

Hypervisor based containers proxy
Apache License 2.0
32 stars 15 forks source link

k8s cannot start pods after revendoring virtcontainers with latest changes #212

Open chavafg opened 6 years ago

chavafg commented 6 years ago

From #196, we started getting k8s issues. cc-runtime start gets hanged:

fuentess@k8s-latest:~/go/src/github.com/clearcontainers/tests/integration/kubernetes$ ps -ef | grep cc-
root      26106      1  0 15:29 ?        00:00:03 /usr/bin/dockerd -D --add-runtime cc-runtime=/usr/local/bin/cc-runtime --default-runtime=cc-runtime --storage-driver=overlay2
root      71829      1  0 15:47 ?        00:00:00 /usr/libexec/clear-containers/cc-proxy -uri unix:///run/virtcontainers/pods/2b3a83ac3d1e99622d3f88e1fb67db6f2a05e6cfcbf9488789cd0c7078ac9772/proxy.sock -log debu
g
root      72457  68574  0 15:47 ?        00:00:00 /usr/local/bin/cc-runtime start 6378f883026ec3c84d3c27fcd2626ae4e813a01eaa55bb6b199db364ed0f9789
root      72480  68574  0 15:47 ?        00:00:00 /usr/local/bin/cc-runtime state fa5dcb4eb9109e50ed0a42462257c1d2408aa200a7418a27f09fce0a52045a8e

From the cc-proxy journal this seems to be the error:

Feb 01 15:47:39 k8s-latest cc-proxy[71829]: time="2018-02-01T15:47:39.262570917Z" level=info msg="waiting for shim to register itself with token yGqVkAtqCysPIthxCxsEfE2OPvVAECkj8UrAPdCsb-M= (timeout 30s)" name=c
c-proxy pid=71829 section=io source=proxy vm=2b3a83ac3d1e99622d3f88e1fb67db6f2a05e6cfcbf9488789cd0c7078ac9772
Feb 01 15:47:39 k8s-latest cc-proxy[71829]: time="2018-02-01T15:47:39.26401198Z" level=debug msg="{\\\"channel\\\":\\\"ctl\\\",\\\"command\\\":\\\"newcontainer\\\",\\\"error\\\":\\\"Only 4088 bytes read out of 4
146 expected (ctl channel)\\\",\\\"level\\\":\\\"error\\\",\\\"msg\\\":\\\"Read failed\\\",\\\"name\\\":\\\"cc-agent\\\",\\\"pid\\\":170,\\\"time\\\":\\\"2018-02-01T15:47:39.23652371Z\\\"}" name=cc-proxy pid=718
29 source=qemu vm=2b3a83ac3d1e99622d3f88e1fb67db6f2a05e6cfcbf9488789cd0c7078ac9772
Feb 01 15:47:39 k8s-latest cc-proxy[71829]: time="2018-02-01T15:47:39.289399993Z" level=debug msg="Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1 i
nterface=org.freedesktop.systemd1.Manager member=PowerOff cookie=1 reply_cookie=0 error=n/a" name=cc-proxy pid=71829 source=qemu vm=2b3a83ac3d1e99622d3f88e1fb67db6f2a05e6cfcbf9488789cd0c7078ac9772
Feb 01 15:47:39 k8s-latest cc-proxy[71829]: time="2018-02-01T15:47:39.290029021Z" level=debug msg="Got message type=method_return sender=n/a destination=n/a object=n/a interface=n/a member=n/a cookie=1 reply_coo
kie=1 error=n/a" name=cc-proxy pid=71829 source=qemu vm=2b3a83ac3d1e99622d3f88e1fb67db6f2a05e6cfcbf9488789cd0c7078ac9772

@amshinde suggested to test #196 with latest code on the clearcontainers/agent repository, which seems to have a fix for the error above. But latest code from agent also has issues when creating pods: https://github.com/clearcontainers/agent/issues/182