Open kmahelona opened 1 year ago
Hi @kmahelona thanks for the details and sorry to hear you are experiencing this. Unfortunately my surface level research into this problem is coming up short. Some things that might help debug:
Have you always experienced this crash with this tool or is it more recent? Anything particular you have done differently recently that could help give hints?
I've only used the tool for this purpose.
Is the crash only reproducible when making requests through the nginx proxy? Can you reproduce when hitting the respective containers directly?
I don't seem to be getting crashes with nginx proxy without mac-docker-connect.
I've replicated the crashes taking nginx out of the loop, and connecting directly to the services. In my test the only static IP I connected to is the one associated with the docker running the web sockets.
Any notably large number of requests being made during the crash? I noticed you have a Web-socket server. Any correlation between the crash and making connections to that?
I manually make a lot of requests to a page that doesn't have web-socket connections, and I can't seem to crash Docker. Once I navigate to a page with web-sockets, the docker may crash. If it doesn't crash I keep refreshing until it does. I was suspecting that the container running the web-socket (vue hot reloading with an inspector plugin that uses web-sockets) was part of the problem and the test above further supports this.
Further testing I can confirm:
I should note that there is a request being returned from the docker that's about 28mb in size (a very large vendor js package) but that doesn't cause crashing without docker-mac-connect. Further investigation of this large file,
Without static IP my response times are 300ms
With the static IP my response times are 3s
And when I generated a crash, we can see that the large file didn't load... so that must be part of the problem!
I note that docker-mac-connect was running when I generated the snapshots above, so simply running in the background didn't cause a crash.
Are you on an Intel based Mac or Apple silicon (M1, M2)?
M1
Hey @gregnr great work you’ve done here with the tool and I learned a bunch from its implementation, thank you!
About this issue here, have you thought that it might be somehow related to the MTU size? I’m not sure how that would lead to a complete crash of the VM where docker is running though. I assume that the way VPNKit is forwarding traffic (if you simply forward a port to host) ends up that the host OS gets the MTU size as if there’s no VPNKit in between (which is 64KB for loopback), but with WireGuard you’re at the default of 1420 bytes. Given that WireGuard client runs in the VM as a kernel module, that might make it more plausible that a crash in it would lead to a crash of the kernel? I assume that WireGuard is forced to split the TCP packet that it gets from the container and maybe its internal buffer isn’t able to handle that much? (could also very well be a bug in wg!)
Summary
I've been experiencing random crashes and have tried to figure out what the cause was. I noticed the crash happens when I refresh my site. Since uninstalling docker-mac-net-connect the problem has gone away, but it's not fair to say it was indeed the culprit and I would appreciate some assistance to replicate and find the cause.
Logs
Logs were captured with,
Discussion
When the crash occurs, this is the output I get around the same time,
Earlier I thought this was the culprit but upon replicating the problem there wasn't a crash afte this (as seen in logs above),
I was wondering is it a memory or CPU issue... I've given docker 32GB ram and 8 CPUs so I can't see how that could be the problem. My setup I've modified
/etc/hosts
to point a domain name to a docker running nginx listening on port 80, 443, 8003. I then forward from nginx to other dockers, doing TLS termination etc.OS & Version Docker 4.14.1 (91661) MacOS Monterey 12.6
Being able to bind dockers to different IPs is a great feature for local development when you're working accross a number of different projects, so I very much appreciate this project!