virtio-vsock can't handle a high connection rate

[djs55]

This Docker for Mac bug report: https://github.com/docker/for-mac/issues/1417 contains an interesting set of repro steps:

dd if=/dev/zero of=zero.txt bs=1000 count=1
docker run -dit -p 8888:80 --name apache -v "$PWD":/usr/local/apache2/htdocs/ httpd:2.4
for i in {1..100}; do curl -s -S -0 --no-keepalive "http://127.0.0.1:8888/zero.txt?[1-100]" > /dev/null & done; wait

I believe this runs 100 concurrent instances of curl which each perform 100 TCP opens and closes in series. If I run this program inside Moby (where 127.0.0.1:8888 is also bound by the user space proxy) then it works fine.

If I run it on the Mac, I get a bunch of errors about dropped connections.

The difference between the 2 configurations is on the Mac, vpnkit is listening on the TCP port and calling connect to the Unix domain socket used to establish virtio-vsock connections. It then forwards to this Unix domain socket.

I don't see any sign of hitting the vpnkit maximum connection limit (although if the fds weren't closed promptly then the host could be temporarily running out of fds). Ideally I'd like to run a test on the Mac which bypassed the virtio-vsock interface but I don't have anything set up atm (perhaps using vmnet.framework would do the trick?)

I suspect that the virtio-vsock Unix domain socket interface is unable to cope with the high connect/close rate. If this is a fundamental (or hard to fix) problem then we could rewrite the proxy to multiplex over one virtio-vsock connection.

[ijc]

Does increasing the VTSOCK_MAXSOCKS define help?

While you are rebuilding anyway you may as well set pci_vtsock_debug to 1 of if that is too verbose modify the DPRINT's in connect_sock and handle_connect_fd to be unconditional so you see them (perhaps the code should be more verbose by default on these paths?). Setting pci_vtsock_debug might give more insight in the case where VTSOCK_MAXSOCKS is not the issue though.