Port forward never connects

[gjabell]

Hey all,

I'm working on a project that creates ram-only VMs using Docker. At the moment, we're trying to add support for hyperkit, but we're struggling to get vpnkit to properly forward ports to the virtual machine so that we can SSH into it.

Since we already have Docker installed as part of the tool, we're using Docker's hyperkit and vpnkit binaries for the following. First, we start a vpnkit instance with the command:

vpnkit --ethernet sockets/vpnkit.socket --port sockets/vpnkit.port.socket --vsock-path sockets/connect --debug

This seems to successfully create a vpnkit instance, bound to the specified sockets in the working directory.

Next, we spin up a hyperkit VM (in the same directory) using:

hyperkit -A -m 1G -u -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=sockets/vpnkit.socket -s 2,virtio-sock,guest_cid=3,path=sockets -s 3,virtio-rnd -s 4,virtio-9p,path=sockets/vpnkit.port.socket,tag=port -l com1, stdio -f kexec,$KERNEL,$INITRD,"console=ttyS0"

KERNEL and INITRD point to the paths for our kernel and initramfs image, which we create by saving a Docker image and its respective kernel.

Everything works so far, and the VM is able to boot and connect to the internet. Once the VM has booted, we mount the 9p filesystem at /port using:

mount -t 9p -o trans=virtio,version=9p2000 port /port

Then we run the commands for connecting to the vpnkit instance:

vpnkit-forwarder

and

vpnkit-expose-port -host-ip 0.0.0.0 -host-port 2002 -container-ip 127.0.0.1 -container-port 22 -no-local-ip -i

These binaries are built using the vpnkit-forwarder and vpnkit-expose-port Dockerfiles.

Everything still appears to be working at this point; the /port directory shows a new directory called tcp:0.0.0.0:2002:tcp:127.0.0.1:22, the vpnkit debug log shows Created instance tcp:0.0.0.0:2002:tcp:127.0.0.1:22, and the 2002 port shows as LISTENING on the host.

Unfortunately, attempting to SSH into this port using ssh -p 2002 localhost hang indefinitely, with the command never timing out but also never connecting. The vpnkit debug log shows tcp:0.0.0.0:2002:tcp:127.0.0.1:22: connected after the connection is first established, but doesn't offer any logs past that. We can confirm that the SSH daemon is running on the VM, and have also verified that this behavior occurs when forwarding a different port / service (like a web server).

Does anyone have insight into why this might be happening, or how we can further debug this issue?

[djs55]

Hi,

I think you've got all the components necessary so I'm not sure why it's not working. I would recommend using strace on the processes involved, in particular vpnkit-forwarder. Is it receiving connections from the host over hypervisor sockets? If not then the problem might be vpnkit failing to connect to the Unix domain socket that hyperkit is listening on? If so then check out the vpnkit argument

       --vsock-path=VSOCK, --port-forwards=VSOCK
           Path of the Unix domain socket used to setup virtio-vsock connections to the VM.  

Good luck!

[gjabell]

Hey @djs55,

Sorry for the slow response! After some more debugging, it seems that the kernel is to blame here; we can get it to work using one of LinuxKit's kernels but not the default distribution kernel. Unfortunately, that isn't the best option for us, so we have decided to switch to using a virtio-net interface for hyperkit instead of vpnkit.

I haven't had the time to dig into which part of LinuxKit's patches and kernel mods actually cause vpnkit to work properly, but it did appear that vpnkit was failing to connect over the Unix domain socket (in our case, sockets/connect). Docker's vsock-path directory contained a number of file-sockets that I assume facilitated connections between Docker's hyperkit and vpnkit, while our socket directory never contained any additional files.

Anyways, thanks for your help on this issue! I'll go ahead and close it since we're going with a different solution, but if you decide you want to look into it further I can help with debugging.

Galen