Cannot make traffic from host on Mac to guest IP with vpnkit

[wkozaczuk]

I could not find any e-mailing list or another forum I could post this question to so I am creating an issue which I will close eventually.

So I am trying to make inbound connect to the OSv unikernel guest running on hyperkit over vpnkit and it does not seem to work or I simply do not know which host IP address to use.

I am using Docker Desktop community version 2.0.0.4. By default I have 2 processes related to Docker/hyperkit/vpnkit running like so:

ps -ef | grep hyperkit
  501  3258  3256   0 Thu09AM ??         0:25.76 com.docker.vpnkit --ethernet fd:3 --port vpnkit.port.sock --port hyperkit://:62373/./vms/0 --diagnostics fd:4 --pcap fd:5 --vsock-path vms/0/connect --host-names host.docker.internal,docker.for.mac.host.internal,docker.for.mac.localhost --gateway-names gateway.docker.internal,docker.for.mac.gateway.internal,docker.for.mac.http.internal --vm-names docker-for-desktop --listen-backlog 32 --mtu 1500 --allowed-bind-addresses 0.0.0.0 --http /Users/****/Library/Group Containers/group.com.docker/http_proxy.json --dhcp /Users/****/Library/Group Containers/group.com.docker/dhcp.json --port-max-idle-time 300 --max-connections 2000 --gateway-ip 192.168.65.1 --host-ip 192.168.65.2 --lowest-ip 192.168.65.3 --highest-ip 192.168.65.254 --log-destination asl --udpv4-forwards 123:127.0.0.1:53859 --gc-compact-interval 1800
  501  3265  3260   0 Thu09AM ??        22:45.92 com.docker.hyperkit -A -u -F vms/0/hyperkit.pid -c 4 -m 2048M -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-vpnkit,path=vpnkit.eth.sock,uuid=fb0a6ebd-8e87-42c2-ae06-83c1027f0c55 -U 8354e93d-6601-4091-923f-0428d28d1c0c -s 2:0,ahci-hd,/Users/****/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.raw -s 3,virtio-sock,guest_cid=3,path=vms/0,guest_forwards=2376;1525 -s 4,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker-for-mac.iso -s 5,ahci-cd,vms/0/config.iso -s 6,ahci-cd,/Applications/Docker.app/Contents/Resources/linuxkit/docker.iso -s 7,virtio-rnd -l com1,autopty=vms/0/tty,asl -f bootrom,/Applications/Docker.app/Contents/Resources/uefi/UEFI.fd,,

When I start OSv on hyperkit executing to virtio (instead of vpnkit), the OSv starts and grabs 192.168.64.16 (using dhcp) like so:

Invoking HYPERKIT at: /usr/local/bin/hyperkit with arguments:
  -A
  -x
  -c 2
  -m 1024M
  -f kexec,/Users/***/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar
  -l com1,stdio
  -s 0:0,hostbridge
  -s 31,lpc
  -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit
  -s 2:0,virtio-net
OSv v0.53.0-61-g513e776b
eth0: 192.168.64.16
....

and I can make traffic over port 8080 from the host, more specifically the guest responds on HTTP at http://192.168.64.16:8080/

The extra hyperkit process looks like this:

0 19871 19869   0  6:08PM ttys007    0:39.01 /usr/local/bin/hyperkit -A -x -c 2 -m 1024M -f kexec,/Users/****/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-blk,/Users/****/.capstan/repository/spring-boot/spring-boot.hkit -s 2:0,virtio-net

Now when I use vpnkit instead I can see guest grabbing an IP but I cannot connect to it in any way and it does not work.

Invoking HYPERKIT at: /usr/local/bin/hyperkit with arguments:
  -A
  -x
  -c 2
  -m 1024M
  -f kexec,/Users/wkozaczuk/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar
  -l com1,stdio
  -s 0:0,hostbridge
  -s 31,lpc
  -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit
  -s 2:0,virtio-vpnkit,path=/Users/****/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock
  -s 3,virtio-sock,guest_cid=3,path=/Users/***/.capstan/instances/hkit/spring-boot/vsockState
OSv v0.53.0-61-g513e776b
eth0: 192.168.65.24
...

And I see this extra process:

501 19935 19933   0  6:14PM ttys007    0:30.10 /usr/local/bin/hyperkit -A -x -c 2 -m 1024M -f kexec,/Users/***/.capstan/repository/osv-loader/osv-vmlinuz.bin,,/usr/lib/jvm/java/bin/java -Xms64m -Xmx64m -jar spring-boot-2-rest-service-basic.jar -l com1,stdio -s 0:0,hostbridge -s 31,lpc -s 1:0,virtio-blk,/Users/***/.capstan/repository/spring-boot/spring-boot.hkit -s 2:0,virtio-vpnkit,path=/Users/***/Library/Containers/com.docker.docker/Data/vpnkit.eth.sock -s 3,virtio-sock,guest_cid=3,path=/Users/***/.capstan/instances/hkit/spring-boot/vsockState

I have tried obvious 192.168.65.24 as the guest reports grabbing, I have tried host IP 192.168.65.2 and couple of other ones? I event have tried to add ,guest_forwards=8080 to vsock path and also does not work with any IPs.

What am I doing wrong? Please advise.

Waldek

[wkozaczuk]

Any update on this?

[chenkai2]

Same here. I found the privileged helper service which help docker exposes "please bind this port" API, it's /Library/PrivilegedHelperTools/com.docker.vmnetd. But I can't find any info about its protocol to comunicate with dockerd. Now I turn into looking for tunnel solutions like docker mac net connect With a tunnel to physically connect from Host to Docker VM (by default, it's 10.33.33.2), I can port forward with a reverse proxy. In my situation, I'm working with Docker Swarm which is poorly supported by Docker Desktop For Mac, but the ports Swarm Cluster published can only accessed by Docker VM, not Host (Mac OS).