MacOS DNS regression in 1.5.0 and 1.5.1

ryfow commented 2 years ago

Actual Behavior

With Rancher Desktop 1.5.{0,1} on aarch64 MacOS, I'm seeing qemu-system-aarch64 hang for several minutes at a time in my development environment. The problem appears to be triggered by a process making bursty DNS requests for host.docker.internal. The same development environment works fine on Rancher Desktop 1.4.1.

Steps to Reproduce

This isn't how I found the problem, but I think it reproduces the same underlying issue.

Install Rancher Desktop 1.5.1 and configure in Docker/moby mode.
Run docker run --rm --name crashy-crashy -ti ubuntu:20.04 bash -c 'apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y dnsutils psmisc && while true ; do dig host.docker.internal ; done'
Wait for the crashy-crashy container to start logging dig output
Run the following command in another host terminal: docker exec -ti crashy-crashy bash -c "while true ; do killall dig ; sleep .1 ; done"
Wait for a bit, you should see the dig output eventually stop, qemu-system-aarch64 will be running at 100% CPU on your MacOS host, and docker commands will no longer work.

Result

I'm seeing the Rancher Desktop qemu VM become unresponsive until I kill the qemu-system-aarch64 process and restart Rancher Desktop.

Expected Behavior

The Rancher Desktop VM should not hang.

Additional Information

No response

Rancher Desktop Version

1.5.1

Rancher Desktop K8s Version

N/A

Which container engine are you using?

moby (docker cli)

What operating system are you using?

macOS

Operating System / Build Version

MacOS Monterey 12.5.1

What CPU architecture are you using?

arm64 (Apple Silicon)

Linux only: what package format did you use to install Rancher Desktop?

N/A

Windows User Only

No response

ryfow commented 2 years ago

FWIW, I ran my reproduction steps on a second Macbook, and the see the same behavior.

Nino-K commented 1 year ago

FWIW, I ran my reproduction steps on a second Macbook, and they see the same behavior.

@ryfow is the second Macbook also a M1? or x86?

@jandubois do you think you can reproduce this on your M1 machine?

matsukaz commented 1 year ago

Hi, I'm facing to a similar issue in 1.7.0. It seems like Lima is stuck on the file descriptor limit, but I haven't found a way to solve it yet. This issue also occurs on x86 macOS.

Steps to Reproduce

Login to Lima and keep running nslookup.

$ rdctl shell
lima-rancher-desktop:/Users/xxx$ while true; do nslookup www.google.co.jp; done

On host OS, show a list of UDP open files that qemu-system-aarch64 handles.

$ lsof -p $(pgrep qemu-system-aarch64) | grep "UDP"
qemu-syst 6788 xxxx  119u  IPv4 0x2c6ecf140850ff5f         0t0                 UDP *:63544
qemu-syst 6788 xxxx  120u  IPv4 0x2c6ecf140851762f         0t0                 UDP *:63398

A number of UDP open files are keep increasing and after it reaches to FD=1024u, Lima get stuck.

$ lsof -p $(pgrep qemu-system-aarch64) | grep "UDP"
...
qemu-syst 6788 xxxx  1023u  IPv4 0x2c6ecf14085191bf         0t0                 UDP *:54486
qemu-syst 6788 xxxx  1024u  IPv4 0x2c6ecf140852088f         0t0                 UDP *:62934

If you wait exactly 4 minutes, all UDP open files get released and Lima starts running again.

Rancher Desktop Version

1.4.1, 1.6.2, 1.7.0

Rancher Desktop K8s Version

N/A

Which container engine are you using?

moby (docker cli)

Operating System / Build Version / CPU

MacOS Monterey 12.6 (M1 2020) MacOS Ventura 13.0.1 (Intel Core i5, 2019)

matsukaz commented 1 year ago

This Issue may be a problem about Alpine Linux. I tried it with Lima and got the same problem, also with Debian, but not with Ubuntu.

I used the following images.

Nino-K commented 1 year ago

@ryfow the issue has been addressed here: https://github.com/lima-vm/lima/issues/1285, therefore, it should be included in our upcoming release. Thank you again for reporting this.