Unable to use runq runtime in nested VM (VMWareFusion)

[aandryashin]

Brief description It is not possible to use runq docker runtime in nested virtual machine with vmx enabled.

Steps to reproduce the issue Run nested VM with runq docker runtime enabled and try to run any container.

Expected behaviour

$ docker run --runtime runq -ti hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

Actual behaviour

$ docker run --runtime runq -ti hello-world
2018-06-11T13:18:13.114046Z qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0
qemu-system-x86_64: /build/qemu-YSlZyK/qemu-2.11+dfsg/target/i386/kvm.c:1799: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
[proxy v20180228-19-g1c955fc-dirty] conn.Read: read unix /dev/runq.sock->@: read: connection reset by peer
[proxy v20180228-19-g1c955fc-dirty] signal: aborted (core dumped)

Content of section runtimes of /etc/docker/daemon.json

{
  "runtimes": {
    "runq": {
      "path": "/var/lib/runq/runq",
      "runtimeArgs": [
        "--cpu", "1",
        "--mem", "256",
        "--dns", "8.8.8.8,8.8.4.4"
      ]
    }
  }
}

Content of /var/lib/runq/qemu/proxy --version

v20180228-19-g1c955fc-dirty (go1.10.1)

Content of docker --version

Docker version 18.05.0-ce, build f150324

Additional information There was the same issue in Clear Containers: https://github.com/intel/cc-oci-runtime/issues/1053 that was fixed by this commit: https://github.com/containers/virtcontainers/commit/dd13e0b7e0a4957a6b8e982356e0cb4bc227e391#diff-82527fac73a6f9f5442c1db98ce1784aR589

I can confirm that the issue is disappeared when I used "-cpu", "host,pmu=off" option and added disable-modern=true to virtio devices.

[pmorjan]

Thanks for the report. I can recreate the issue. On my MacOS system (quite outdated: El Capitan, Fusion 7.1.3, Fedora 28) "pmu=off" solves the issue. Is "disable-modern=on" on virtio devices really necessary?

[aandryashin]

I am not sure about all virtio devices, may be it is necessary only for root_fs, without that option 9p filesystem cannot be mounted and just hangs.

[pmorjan]

The branch nestedvm contains a new runtime configuration option --nestedvm for testing. It adds the extra qemu command line parameters.

[pmorjan]

The "--nestedvm" option (now in master) also works for Vmware ESXi guests. Setup instructions can be found here: https://communities.vmware.com/docs/DOC-8970