after shim connect failure, cannot stop container via CRICTL stop

[egernst]

issue observed from Cheng Xu:

When “crictl attach” to a fedora container, I tried to “yum install” some package. It hangs, which makes sense since there might be a network connection issue on that machine. But the problem is, after a while I see below error, and after that, “crictl stop” won’t be able to stop the container. Any idea? The repro is pretty consistent, I hit this when demoing last week on my office machine, and also today on an azureVM. Thanks.

handle_proxy_response:616:Error response received from proxy at /run/virtcontainers/pods/e6ee288a536728fdc06399be7555f951627ec71234a12f4b92ae9796fd593d91/proxy.sock: {"msg":"vm: unknown token yEXx89gDGN_6jce7V9g-xrDB706CpuMSjJYm1L4SKYQ="} /usr/libexec/clear-containers/cc-shim: Shim received an error in responseto ConnectShim command,exiting

[egernst]

cc/ @sboeuf @jodh-intel @jcvenegas

[sboeuf]

Nothing in particular comes to mind here since this could be caused by a bunch of different reasons. We'd need to get someone investigating this, being able to reproduce the behavior and debug.

[egernst]

cc/ @chenxumsft

[amshinde]

I'd like to know what storage driver is being used here. We have had issues with yum and 9pfs before. cc @devimc

[chenxumsft]

the crio daemon was configured to use "overlay" as storage driver.. but I guess 9pfs is used for container rootfs?

[egernst]

@chenxumsft - correct. As a quick test, can you change the storage driver to be devicemapper based?

[chenxumsft]

It worked by changing crio to use devicemapper as storage-driver. Thanks. Earlier I ever tried using devicemapper but hit a failure in starting container. but today StartContainer seems working..

[amshinde]

@chenxumsft Glad to hear devicemapper worked out. @egernst Another of our 9pfs issues, I think qemu 2.9 carries a patch that fixes the yum issue iirc. @devimc Can you confirm?

[devimc]

@amshinde that's right