DHCP leases file not getting updated

spowelljr commented 1 year ago

We've been getting issues on our personal machines where our DHCP leases file is not being updated after starting an instance with socket_vmnet. The fix so far has been uninstall socket_vmnet, reboot machine, reinstall, which resolves the issue. However I'm trying to integrate GitHub Action tests for QEMU/socket_vmnet and I'm getting the same thing (logs below).

Is there a command we could run that would resolve this issue? I see in the README there's a command to Reload the DHCP daemon

sudo /bin/launchctl kickstart -kp system/com.apple.bootpd

I'm not sure if that should resolve the issue, however trying to run it in GitHub Action resulted in Could not find service "com.apple.bootpd" in domain for system

Below are to logs from the GitHub Action machine with the main error being StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory

We experienced the same thing on our personal machines, we tried manually creating the file just incase the there were some permission errors but that didn't resolve the issue, plus I'm sure it's more than capable of creating the file itself.

Any help would be appreciated, thanks!

functional_test.go:2161: (dbg) Run:  ./minikube-darwin-amd64 start -p func-184505 --memory=4000 --apiserver-port=8441 --wait=all --driver=qemu
    functional_test.go:2161: (dbg) Non-zero exit: ./minikube-darwin-amd64 start -p func-184505 --memory=4000 --apiserver-port=8441 --wait=all --driver=qemu: exit status 80 (2m23.88093201s)

        -- stdout --
            * [func-184505] minikube v1.28.0 on Darwin 12.6.1
              - MINIKUBE_HOME=/Users/runner/work/minikube/minikube/minikube_binaries/testhome
              - KUBECONFIG=/Users/runner/work/minikube/minikube/minikube_binaries/testhome/kubeconfig
            * Using the qemu2 (experimental) driver based on user configuration
            * Automatically selected the socket_vmnet network
            * Downloading VM boot image ...
            * Starting control plane node func-184505 in cluster func-184505
            * Downloading Kubernetes v1.25.3 preload ...
            * Creating qemu2 VM (CPUs=2, Memory=[40](https://github.com/spowelljr/minikube/actions/runs/3604165929/jobs/6073444435#step:12:41)00MB, Disk=20000MB) ...
            * Deleting "func-184505" in qemu2 ...
            * Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...

        -- /stdout --
        ** stderr ** 
            ! Local proxy ignored: not passing HTTP_PROXY=localhost:49253 to docker env.
            ! Local proxy ignored: not passing HTTP_PROXY=localhost:49253 to docker env.
            ! StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory
            ! Local proxy ignored: not passing HTTP_PROXY=localhost:49253 to docker env.
            * Failed to start qemu2 VM. Running "minikube delete -p func-18[45](https://github.com/spowelljr/minikube/actions/runs/3604165929/jobs/6073444435#step:12:46)05" may fix it: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory
            X Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: open /var/db/dhcpd_leases: no such file or directory

medyagh commented 1 year ago

any idea @AkihiroSuda what might cause this? we see this error as a Flake on our mac Os sometimes, but on Github Action it happens all the time. so I think finding the root cause would help a lot of users

AkihiroSuda commented 1 year ago

Could you make sure that you are NOT using the bridged mode?

Github Action

Bridged mode is known not to work on Github Action

spowelljr commented 1 year ago

I'm just running

sudo make PREFIX=/opt/socket_vmnet install

Which according to the comments in the Makefile Empty value (default) disables bridged mode. bridged mode is not being used.

medyagh commented 1 year ago

@AkihiroSuda are there any debugging commands you suggest I could run to get you more logs? I currently have the problem on Mac M1 with minikube here are the logs if it helps with debugging (the log is too long I attach here) lastStart.txt

but here is the last part of the log


I0103 14:33:10.388012    8417 main.go:134] libmachine: STDERR: 
I0103 14:33:10.388025    8417 main.go:134] libmachine: DONE writing to /Users/medya/.minikube/machines/minikube/disk.qcow2.raw and /Users/medya/.minikube/machines/minikube/disk.qcow2
I0103 14:33:10.388028    8417 main.go:134] libmachine: Starting QEMU VM...
I0103 14:33:10.388082    8417 main.go:134] libmachine: executing: /opt/socket_vmnet/bin/socket_vmnet_client /var/run/socket_vmnet qemu-system-aarch64 -M virt -cpu host -drive file=/opt/homebrew/Cellar/qemu/7.1.0/share/qemu/edk2-aarch64-code.fd,readonly=on,format=raw,if=pflash -display none -accel hvf -m 4000 -smp 2 -boot d -cdrom /Users/medya/.minikube/machines/minikube/boot2docker.iso -qmp unix:/Users/medya/.minikube/machines/minikube/monitor,server,nowait -pidfile /Users/medya/.minikube/machines/minikube/qemu.pid -device virtio-net-pci,netdev=net0,mac=6e:a5:b1:18:4b:c6 -netdev socket,id=net0,fd=3 -daemonize /Users/medya/.minikube/machines/minikube/disk.qcow2
I0103 14:33:10.554509    8417 main.go:134] libmachine: STDOUT: 
I0103 14:33:10.554525    8417 main.go:134] libmachine: STDERR: 
I0103 14:33:10.554530    8417 main.go:134] libmachine: Attempt 0
I0103 14:33:10.554541    8417 main.go:134] libmachine: Searching for 6e:a5:b1:18:4b:c6 in /var/db/dhcpd_leases ...
I0103 14:33:10.555184    8417 main.go:134] libmachine: Found 29 entries in /var/db/dhcpd_leases!
I0103 14:33:10.555206    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.30 HWAddress:be:cf:36:32:32:50 ID:1,be:cf:36:32:32:50 Lease:0x638fe0ed}
I0103 14:33:10.555218    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.29 HWAddress:fa:f0:51:fe:f7:1a ID:1,fa:f0:51:fe:f7:1a Lease:0x638bb50d}
I0103 14:33:10.555223    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.28 HWAddress:6e:89:68:eb:29:5d ID:1,6e:89:68:eb:29:5d Lease:0x638bb494}
I0103 14:33:10.555229    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.27 HWAddress:ca:fb:db:5f:2d:24 ID:1,ca:fb:db:5f:2d:24 Lease:0x638bb19f}
I0103 14:33:10.555242    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.26 HWAddress:d2:3e:a3:7b:7d:f2 ID:1,d2:3e:a3:7b:7d:f2 Lease:0x638ba52c}
I0103 14:33:10.555248    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.25 HWAddress:6e:35:16:67:6f:a8 ID:1,6e:35:16:67:6f:a8 Lease:0x638b9adf}
I0103 14:33:10.555253    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.24 HWAddress:fa:d8:27:37:78:5a ID:1,fa:d8:27:37:78:5a Lease:0x638b9788}
I0103 14:33:10.555259    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.23 HWAddress:46:94:f8:3a:f9:c1 ID:1,46:94:f8:3a:f9:c1 Lease:0x638a7665}
I0103 14:33:10.555266    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.22 HWAddress:6:ac:ca:d6:bc:d0 ID:1,6:ac:ca:d6:bc:d0 Lease:0x6374114f}
I0103 14:33:10.555271    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.21 HWAddress:f2:b0:8:d2:81:81 ID:1,f2:b0:8:d2:81:81 Lease:0x637560c7}
I0103 14:33:10.555276    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.20 HWAddress:56:4b:62:8e:98:7 ID:1,56:4b:62:8e:98:7 Lease:0x63753f29}
I0103 14:33:10.555281    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.19 HWAddress:e6:86:66:d0:9e:39 ID:1,e6:86:66:d0:9e:39 Lease:0x636ec84f}
I0103 14:33:10.555287    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.18 HWAddress:6a:18:75:f:19:7a ID:1,6a:18:75:f:19:7a Lease:0x636ec348}
I0103 14:33:10.555292    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.17 HWAddress:26:34:18:84:98:cb ID:1,26:34:18:84:98:cb Lease:0x636eb11c}
I0103 14:33:10.555307    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.16 HWAddress:ee:25:74:7a:65:f6 ID:1,ee:25:74:7a:65:f6 Lease:0x636eb03f}
I0103 14:33:10.555313    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.15 HWAddress:9a:70:93:66:fa:b5 ID:1,9a:70:93:66:fa:b5 Lease:0x636ea065}
I0103 14:33:10.555318    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.14 HWAddress:96:25:4e:67:c7:fb ID:1,96:25:4e:67:c7:fb Lease:0x636ea026}
I0103 14:33:10.555324    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.13 HWAddress:3a:19:ee:fb:ee:a2 ID:1,3a:19:ee:fb:ee:a2 Lease:0x636e9bc8}
I0103 14:33:10.555333    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.12 HWAddress:3a:dd:b7:8c:e0:7f ID:1,3a:dd:b7:8c:e0:7f Lease:0x636e9519}
I0103 14:33:10.555339    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.11 HWAddress:d2:4e:50:2c:3f:9d ID:1,d2:4e:50:2c:3f:9d Lease:0x636d910e}
I0103 14:33:10.555344    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.10 HWAddress:e2:e5:a7:65:9e:fd ID:1,e2:e5:a7:65:9e:fd Lease:0x636d8e99}
I0103 14:33:10.555349    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.9 HWAddress:e:75:98:a:a4:5b ID:1,e:75:98:a:a4:5b Lease:0x636d8a61}
I0103 14:33:10.555354    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.8 HWAddress:7a:f3:e3:a6:e5:42 ID:1,7a:f3:e3:a6:e5:42 Lease:0x636d88fd}
I0103 14:33:10.555359    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.7 HWAddress:3a:36:74:65:54:48 ID:1,3a:36:74:65:54:48 Lease:0x636d7a5c}
I0103 14:33:10.555365    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.6 HWAddress:be:1a:aa:7c:bb:71 ID:1,be:1a:aa:7c:bb:71 Lease:0x636d7183}
I0103 14:33:10.555370    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.5 HWAddress:c6:c1:8d:ef:8c:6a ID:1,c6:c1:8d:ef:8c:6a Lease:0x636d7108}
I0103 14:33:10.555375    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.4 HWAddress:c2:79:20:b8:68:de ID:1,c2:79:20:b8:68:de Lease:0x636d3d9c}
I0103 14:33:10.555390    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.3 HWAddress:76:f5:fd:cf:d6:1 ID:1,76:f5:fd:cf:d6:1 Lease:0x636c6368}
I0103 14:33:10.555395    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.2 HWAddress:6a:c5:89:c1:b8:15 ID:1,6a:c5:89:c1:b8:15 Lease:0x636c6200}
I0103 14:33:12.556467    8417 main.go:134] libmachine: Attempt 1
I0103 14:33:12.556499    8417 main.go:134] libmachine: Searching for 6e:a5:b1:18:4b:c6 in /var/db/dhcpd_leases ...
I0103 14:33:12.556706    8417 main.go:134] libmachine: Found 29 entries in /var/db/dhcpd_leases!
I0103 14:33:12.556731    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.30 HWAddress:be:cf:36:32:32:50 ID:1,be:cf:36:32:32:50 Lease:0x638fe0ed}
I0103 14:33:12.556738    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.29 HWAddress:fa:f0:51:fe:f7:1a ID:1,fa:f0:51:fe:f7:1a Lease:0x638bb50d}
I0103 14:33:12.556744    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.28 HWAddress:6e:89:68:eb:29:5d ID:1,6e:89:68:eb:29:5d Lease:0x638bb494}
I0103 14:33:12.556751    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.27 HWAddress:ca:fb:db:5f:2d:24 ID:1,ca:fb:db:5f:2d:24 Lease:0x638bb19f}
I0103 14:33:12.556757    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.26 HWAddress:d2:3e:a3:7b:7d:f2 ID:1,d2:3e:a3:7b:7d:f2 Lease:0x638ba52c}
I0103 14:33:12.556763    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.25 HWAddress:6e:35:16:67:6f:a8 ID:1,6e:35:16:67:6f:a8 Lease:0x638b9adf}
I0103 14:33:12.556769    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.24 HWAddress:fa:d8:27:37:78:5a ID:1,fa:d8:27:37:78:5a Lease:0x638b9788}
I0103 14:33:12.556775    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.23 HWAddress:46:94:f8:3a:f9:c1 ID:1,46:94:f8:3a:f9:c1 Lease:0x638a7665}
I0103 14:33:12.556783    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.22 HWAddress:6:ac:ca:d6:bc:d0 ID:1,6:ac:ca:d6:bc:d0 Lease:0x6374114f}
I0103 14:33:12.556789    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.21 HWAddress:f2:b0:8:d2:81:81 ID:1,f2:b0:8:d2:81:81 Lease:0x637560c7}
I0103 14:33:12.556795    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.20 HWAddress:56:4b:62:8e:98:7 ID:1,56:4b:62:8e:98:7 Lease:0x63753f29}
I0103 14:33:12.556801    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.19 HWAddress:e6:86:66:d0:9e:39 ID:1,e6:86:66:d0:9e:39 Lease:0x636ec84f}
I0103 14:33:12.556806    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.18 HWAddress:6a:18:75:f:19:7a ID:1,6a:18:75:f:19:7a Lease:0x636ec348}
I0103 14:33:12.556812    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.17 HWAddress:26:34:18:84:98:cb ID:1,26:34:18:84:98:cb Lease:0x636eb11c}
I0103 14:33:12.556819    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.16 HWAddress:ee:25:74:7a:65:f6 ID:1,ee:25:74:7a:65:f6 Lease:0x636eb03f}
I0103 14:33:12.556825    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.15 HWAddress:9a:70:93:66:fa:b5 ID:1,9a:70:93:66:fa:b5 Lease:0x636ea065}
I0103 14:33:12.556831    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.14 HWAddress:96:25:4e:67:c7:fb ID:1,96:25:4e:67:c7:fb Lease:0x636ea026}
I0103 14:33:12.556837    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.13 HWAddress:3a:19:ee:fb:ee:a2 ID:1,3a:19:ee:fb:ee:a2 Lease:0x636e9bc8}
I0103 14:33:12.556843    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.12 HWAddress:3a:dd:b7:8c:e0:7f ID:1,3a:dd:b7:8c:e0:7f Lease:0x636e9519}
I0103 14:33:12.556852    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.11 HWAddress:d2:4e:50:2c:3f:9d ID:1,d2:4e:50:2c:3f:9d Lease:0x636d910e}
I0103 14:33:12.556858    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.10 HWAddress:e2:e5:a7:65:9e:fd ID:1,e2:e5:a7:65:9e:fd Lease:0x636d8e99}
I0103 14:33:12.556864    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.9 HWAddress:e:75:98:a:a4:5b ID:1,e:75:98:a:a4:5b Lease:0x636d8a61}
I0103 14:33:12.556870    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.8 HWAddress:7a:f3:e3:a6:e5:42 ID:1,7a:f3:e3:a6:e5:42 Lease:0x636d88fd}
I0103 14:33:12.556876    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.7 HWAddress:3a:36:74:65:54:48 ID:1,3a:36:74:65:54:48 Lease:0x636d7a5c}
I0103 14:33:12.556882    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.6 HWAddress:be:1a:aa:7c:bb:71 ID:1,be:1a:aa:7c:bb:71 Lease:0x636d7183}
I0103 14:33:12.556888    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.5 HWAddress:c6:c1:8d:ef:8c:6a ID:1,c6:c1:8d:ef:8c:6a Lease:0x636d7108}
I0103 14:33:12.556894    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.4 HWAddress:c2:79:20:b8:68:de ID:1,c2:79:20:b8:68:de Lease:0x636d3d9c}
I0103 14:33:12.556900    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.3 HWAddress:76:f5:fd:cf:d6:1 ID:1,76:f5:fd:cf:d6:1 Lease:0x636c6368}
I0103 14:33:12.556906    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.2 HWAddress:6a:c5:89:c1:b8:15 ID:1,6a:c5:89:c1:b8:15 Lease:0x636c6200}
I0103 14:33:14.557930    8417 main.go:134] libmachine: Attempt 2
I0103 14:33:14.557940    8417 main.go:134] libmachine: Searching for 6e:a5:b1:18:4b:c6 in /var/db/dhcpd_leases ...
I0103 14:33:14.558076    8417 main.go:134] libmachine: Found 29 entries in /var/db/dhcpd_leases!
I0103 14:33:14.558084    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.30 HWAddress:be:cf:36:32:32:50 ID:1,be:cf:36:32:32:50 Lease:0x638fe0ed}
I0103 14:33:14.558088    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.29 HWAddress:fa:f0:51:fe:f7:1a ID:1,fa:f0:51:fe:f7:1a Lease:0x638bb50d}
I0103 14:33:14.558093    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.28 HWAddress:6e:89:68:eb:29:5d ID:1,6e:89:68:eb:29:5d Lease:0x638bb494}
I0103 14:33:14.558097    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.27 HWAddress:ca:fb:db:5f:2d:24 ID:1,ca:fb:db:5f:2d:24 Lease:0x638bb19f}
I0103 14:33:14.558101    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.26 HWAddress:d2:3e:a3:7b:7d:f2 ID:1,d2:3e:a3:7b:7d:f2 Lease:0x638ba52c}
I0103 14:33:14.558104    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.25 HWAddress:6e:35:16:67:6f:a8 ID:1,6e:35:16:67:6f:a8 Lease:0x638b9adf}
I0103 14:33:14.558108    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.24 HWAddress:fa:d8:27:37:78:5a ID:1,fa:d8:27:37:78:5a Lease:0x638b9788}
I0103 14:33:14.558112    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.23 HWAddress:46:94:f8:3a:f9:c1 ID:1,46:94:f8:3a:f9:c1 Lease:0x638a7665}
I0103 14:33:14.558116    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.22 HWAddress:6:ac:ca:d6:bc:d0 ID:1,6:ac:ca:d6:bc:d0 Lease:0x6374114f}
I0103 14:33:14.558129    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.21 HWAddress:f2:b0:8:d2:81:81 ID:1,f2:b0:8:d2:81:81 Lease:0x637560c7}
I0103 14:33:14.558133    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.20 HWAddress:56:4b:62:8e:98:7 ID:1,56:4b:62:8e:98:7 Lease:0x63753f29}
I0103 14:33:14.558137    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.19 HWAddress:e6:86:66:d0:9e:39 ID:1,e6:86:66:d0:9e:39 Lease:0x636ec84f}
I0103 14:33:14.558141    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.18 HWAddress:6a:18:75:f:19:7a ID:1,6a:18:75:f:19:7a Lease:0x636ec348}
I0103 14:33:14.558145    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.17 HWAddress:26:34:18:84:98:cb ID:1,26:34:18:84:98:cb Lease:0x636eb11c}
I0103 14:33:14.558149    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.16 HWAddress:ee:25:74:7a:65:f6 ID:1,ee:25:74:7a:65:f6 Lease:0x636eb03f}
I0103 14:33:14.558153    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.15 HWAddress:9a:70:93:66:fa:b5 ID:1,9a:70:93:66:fa:b5 Lease:0x636ea065}
I0103 14:33:14.558156    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.14 HWAddress:96:25:4e:67:c7:fb ID:1,96:25:4e:67:c7:fb Lease:0x636ea026}
I0103 14:33:14.558160    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.13 HWAddress:3a:19:ee:fb:ee:a2 ID:1,3a:19:ee:fb:ee:a2 Lease:0x636e9bc8}
I0103 14:33:14.558164    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.12 HWAddress:3a:dd:b7:8c:e0:7f ID:1,3a:dd:b7:8c:e0:7f Lease:0x636e9519}
I0103 14:33:14.558168    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.11 HWAddress:d2:4e:50:2c:3f:9d ID:1,d2:4e:50:2c:3f:9d Lease:0x636d910e}
I0103 14:33:14.558172    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.10 HWAddress:e2:e5:a7:65:9e:fd ID:1,e2:e5:a7:65:9e:fd Lease:0x636d8e99}
I0103 14:33:14.558179    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.9 HWAddress:e:75:98:a:a4:5b ID:1,e:75:98:a:a4:5b Lease:0x636d8a61}
I0103 14:33:14.558183    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.8 HWAddress:7a:f3:e3:a6:e5:42 ID:1,7a:f3:e3:a6:e5:42 Lease:0x636d88fd}
I0103 14:33:14.558187    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.7 HWAddress:3a:36:74:65:54:48 ID:1,3a:36:74:65:54:48 Lease:0x636d7a5c}
I0103 14:33:14.558191    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.6 HWAddress:be:1a:aa:7c:bb:71 ID:1,be:1a:aa:7c:bb:71 Lease:0x636d7183}
I0103 14:33:14.558195    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.5 HWAddress:c6:c1:8d:ef:8c:6a ID:1,c6:c1:8d:ef:8c:6a Lease:0x636d7108}
I0103 14:33:14.558199    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.4 HWAddress:c2:79:20:b8:68:de ID:1,c2:79:20:b8:68:de Lease:0x636d3d9c}
I0103 14:33:14.558203    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.3 HWAddress:76:f5:fd:cf:d6:1 ID:1,76:f5:fd:cf:d6:1 Lease:0x636c6368}
I0103 14:33:14.558207    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.2 HWAddress:6a:c5:89:c1:b8:15 ID:1,6a:c5:89:c1:b8:15 Lease:0x636c6200}
I0103 14:33:16.559250    8417 main.go:134] libmachine: Attempt 3
I0103 14:33:16.559259    8417 main.go:134] libmachine: Searching for 6e:a5:b1:18:4b:c6 in /var/db/dhcpd_leases ...
I0103 14:33:16.559410    8417 main.go:134] libmachine: Found 29 entries in /var/db/dhcpd_leases!
I0103 14:33:16.559418    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.30 HWAddress:be:cf:36:32:32:50 ID:1,be:cf:36:32:32:50 Lease:0x638fe0ed}
I0103 14:33:16.559427    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.29 HWAddress:fa:f0:51:fe:f7:1a ID:1,fa:f0:51:fe:f7:1a Lease:0x638bb50d}
I0103 14:33:16.559432    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.28 HWAddress:6e:89:68:eb:29:5d ID:1,6e:89:68:eb:29:5d Lease:0x638bb494}
I0103 14:33:16.559436    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.27 HWAddress:ca:fb:db:5f:2d:24 ID:1,ca:fb:db:5f:2d:24 Lease:0x638bb19f}
I0103 14:33:16.559440    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.26 HWAddress:d2:3e:a3:7b:7d:f2 ID:1,d2:3e:a3:7b:7d:f2 Lease:0x638ba52c}
I0103 14:33:16.559444    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.25 HWAddress:6e:35:16:67:6f:a8 ID:1,6e:35:16:67:6f:a8 Lease:0x638b9adf}
I0103 14:33:16.559448    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.24 HWAddress:fa:d8:27:37:78:5a ID:1,fa:d8:27:37:78:5a Lease:0x638b9788}
I0103 14:33:16.559459    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.23 HWAddress:46:94:f8:3a:f9:c1 ID:1,46:94:f8:3a:f9:c1 Lease:0x638a7665}
I0103 14:33:16.559464    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.22 HWAddress:6:ac:ca:d6:bc:d0 ID:1,6:ac:ca:d6:bc:d0 Lease:0x6374114f}
I0103 14:33:16.559468    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.21 HWAddress:f2:b0:8:d2:81:81 ID:1,f2:b0:8:d2:81:81 Lease:0x637560c7}
I0103 14:33:16.559474    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.20 HWAddress:56:4b:62:8e:98:7 ID:1,56:4b:62:8e:98:7 Lease:0x63753f29}
I0103 14:33:16.559478    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.19 HWAddress:e6:86:66:d0:9e:39 ID:1,e6:86:66:d0:9e:39 Lease:0x636ec84f}
I0103 14:33:16.559482    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.18 HWAddress:6a:18:75:f:19:7a ID:1,6a:18:75:f:19:7a Lease:0x636ec348}
I0103 14:33:16.559486    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.17 HWAddress:26:34:18:84:98:cb ID:1,26:34:18:84:98:cb Lease:0x636eb11c}
I0103 14:33:16.559490    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.16 HWAddress:ee:25:74:7a:65:f6 ID:1,ee:25:74:7a:65:f6 Lease:0x636eb03f}
I0103 14:33:16.559500    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.15 HWAddress:9a:70:93:66:fa:b5 ID:1,9a:70:93:66:fa:b5 Lease:0x636ea065}
I0103 14:33:16.559504    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.14 HWAddress:96:25:4e:67:c7:fb ID:1,96:25:4e:67:c7:fb Lease:0x636ea026}
I0103 14:33:16.559508    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.13 HWAddress:3a:19:ee:fb:ee:a2 ID:1,3a:19:ee:fb:ee:a2 Lease:0x636e9bc8}
I0103 14:33:16.559512    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.12 HWAddress:3a:dd:b7:8c:e0:7f ID:1,3a:dd:b7:8c:e0:7f Lease:0x636e9519}
I0103 14:33:16.559516    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.11 HWAddress:d2:4e:50:2c:3f:9d ID:1,d2:4e:50:2c:3f:9d Lease:0x636d910e}
I0103 14:33:16.559520    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.10 HWAddress:e2:e5:a7:65:9e:fd ID:1,e2:e5:a7:65:9e:fd Lease:0x636d8e99}
I0103 14:33:16.559524    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.9 HWAddress:e:75:98:a:a4:5b ID:1,e:75:98:a:a4:5b Lease:0x636d8a61}
I0103 14:33:16.559528    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.8 HWAddress:7a:f3:e3:a6:e5:42 ID:1,7a:f3:e3:a6:e5:42 Lease:0x636d88fd}
I0103 14:33:16.559532    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.7 HWAddress:3a:36:74:65:54:48 ID:1,3a:36:74:65:54:48 Lease:0x636d7a5c}
I0103 14:33:16.559536    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.6 HWAddress:be:1a:aa:7c:bb:71 ID:1,be:1a:aa:7c:bb:71 Lease:0x636d7183}
I0103 14:33:16.559540    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.5 HWAddress:c6:c1:8d:ef:8c:6a ID:1,c6:c1:8d:ef:8c:6a Lease:0x636d7108}
I0103 14:33:16.559544    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.4 HWAddress:c2:79:20:b8:68:de ID:1,c2:79:20:b8:68:de Lease:0x636d3d9c}
I0103 14:33:16.559548    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.3 HWAddress:76:f5:fd:cf:d6:1 ID:1,76:f5:fd:cf:d6:1 Lease:0x636c6368}
I0103 14:33:16.559551    8417 main.go:134] libmachine: dhcp entry: {Name:minikube IPAddress:192.168.105.2 HWAddress:6a:c5:89:c1:b8:15 ID:1,6a:c5:89:c1:b8:15 Lease:0x636c6200}
I0103 14:33:18.560554    8417 main.go:134] libmachine: Attempt 4
I0103 14:33:18.560563    8417 main.go:134] libmachine: Searching for 6e:a5:b1:18:4b:c6 in /var/db/dhcpd_leases ...
I0103 14:33:18.560681    8417 main.go:134] libmachine: Found 29 entries in /var/db/dhcpd_leases!

....
.... (and it keeps going like this till failure)

spowelljr commented 1 year ago

When an instance is terminated it logs the following:

$ cat /var/run/socket_vmnet.stderr
on_accept(): vmnet_return_t VMNET_INVALID_ARGUMENT
vmnet_write: Undefined error: 0

Not sure if that's expected or not.

spowelljr commented 1 year ago

@AkihiroSuda Is there anything I could do (debug commands, etc.) to help figure out this issue? I'm still experiencing this issue.

AkihiroSuda commented 1 year ago

Cc @jandubois could you take a look?

spowelljr commented 1 year ago

I just uninstalled, rebooted, and installed with master and it works now, not sure if recent commits fixed this

mprimeaux commented 1 year ago

@spowelljr I assume you built from source? I ask because there is a brew formula for socket_vmnet as well.

spowelljr commented 1 year ago

Correct, from source

mprimeaux commented 1 year ago

Thanks. Would be good to know if it still works after surviving a reboot. Regardless, I'll perform the exact steps you did and update everyone.

mprimeaux commented 1 year ago

TLDR; DHCP address allocation works when not in bridged mode and directly building socket_vmnet from the latest sources. Bridged mode fails.

When compiling for bridged mode with...

sudo make PREFIX=/opt/socket_vmnet install BRIDGED=en0

...and then starting minikube with....

minikube start \
  --network socket_vmnet \
  --socket-vmnet-client-path="/opt/socket_vmnet/bin/socket_vmnet_client" \
  --socket-vmnet-path="/var/run/socket_vmnet.bridged.en0"

...results in the following failure:

😿  Failed to start qemu2 VM. Running "minikube delete" may fix it: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 76:b3:20:a2:9b:74

❌  Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 76:b3:20:a2:9b:74

Of course minikube delete does not 'fix it'.

What's interesting is I can confirm a new DHCP address is being allocated since my firewall notifies me of new device activity. However, I don't see any new addresses in /var/db/dhcp_leases. I can also confirm the MAC address of the newly allocated DHCP address lease matches the MAC address in the above error.

I've also followed this "Known Issues" workaround to no avail.

UPDATE: As another interesting data point, I am able to start Lima using the same socket_vmnet in bridged mode without any issue using the following excerpt from limactl start --name=default template://vmnet.

network:
  - lima: bridged

jandubois commented 1 year ago

However, I don't see any new addresses in /var/db/dhcp_leases.

That is expected, as bridged IP addresses are assigned by the DHCP on your local network, and are not local to your host. That is the whole point of using a bridged network: to get an externally routable IP address, so other machines can connect to your VM.

Maybe this helps:

LimaVdeVmnet

socket_vmnet works basically the same, except there is no vde_switch involved.

mprimeaux commented 1 year ago

@jandubois Yes. That helps quite a bit. Thanks!

mprimeaux commented 1 year ago

@jandubois I've attached the logs.txt from my latest attempt with bridged mode.

Given the logs are Minikube, if folks prefer, I can move this over to the linked Minikube issue 15300.

jandubois commented 1 year ago

@mprimeaux I'm not sure what you are trying to achieve. You will not be able to get the IP address of the bridged network from the host machine. You will have to run e.g. ip a show lima0 (replace with iface name of your bridged network) inside the VM.

It looks like minikube only supports looking in the local leases log, so it does not support bridged networks at the moment. Somebody will have to add it. If you don't have a slirp interface to use in addition to the bridged network, then you will have to add something to the VM setup to log the address in a place that can be inspected externally.

I tried to see if there is a way to determine the address via arp -a, but it seems complicated. The address is not present initially, so I tried running ping 192.168.255.255 (because my local net is 192.168/16), but even after running it for a minute, I don't get a ping response from my VM. I have to ping it directly, but even looping over just a /24 network and pinging each address takes considerable time.

There is probably a way to force the VM to announce itself, e.g. by incorrectly broadcasting that the MAC address belongs to the host, but then giving up when the VM replies "well, actually, that is my MAC address". But that feels rather hacky (and I haven't tried if it actually works, as the bridged interface technically does belong to the host).

In all, querying the IP address via a local slirp connection seems to be the way to do this.

jandubois commented 1 year ago

I can move this over to the linked Minikube issue 15300.

I think this is just missing functionality in minikube and don't see what could be done in the socket_vmnet repo to improve the situation, so the discussion should probably more there.

mprimeaux commented 1 year ago

@jandubois In short, what I'm trying to accomplish is to have the Kubernetes control plane (minikube) obtain a routable address on my host network.

medyagh commented 1 year ago

I still have this problem, I confirm that uninstalling and rebooting and re-installing from source fixes it (but after a few days it goes bad again) so the root of the problem still exists on the code on HEAD.

@AkihiroSuda is there any reason a restart would fix this ? is there a process other than "/opt/socket_vmnet/bin/socket_vmnet" that we could check for, that maybe is not killed or stoppped after installation ? that only a restart kills it ?

or is there a "lock" on a file or something like that ? I just need to find out what exactly "restart" does that uninstalling/reinstalling without restart wont do the trick....

that way we could tell our users or make minikube just do that when it happens automatically

mprimeaux commented 1 year ago

@medyagh It's coincidental that you mention this as this happened to me just today. I ended up rebooting to fix. Could this be due to network going to sleep or an other power option in macOS? Just a hypothesis.

AkihiroSuda commented 1 year ago

My assumption is that this is an issue of Apple's bootpd (https://github.com/apple-oss-distributions/bootp) or com.apple.NetworkSharing , but not 100% sure.

If sudo /bin/launchctl kickstart -kp system/com.apple.bootpd doesn't work, sudo /bin/launchctl kickstart -kp system/com.apple.NetworkSharing may potentially work. Restarting other services (see sudo /bin/launchctl list) might be worth trying too.

medyagh commented 1 year ago

thanks for giving somethings to try out, they dont seem to fix it but i apperciate that you try

here are the logs just in case:

$ minikube start --driver=qemu2
😄  minikube v1.29.0 on Darwin 13.2 (arm64)
✨  Using the qemu2 driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for d2:c3:e7:45:fe:7d
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...\ ^C

$ minikube delete --all
🔥  Deleting "minikube" in qemu2 ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles

$ sudo /bin/launchctl kickstart -kp system/com.apple.bootpd
Could not find service "com.apple.bootpd" in domain for system
$ sudo /bin/launchctl kickstart -kp system/com.apple.NetworkSharing
service spawned with pid: 4842

$ minikube start --driver=qemu2
😄  minikube v1.29.0 on Darwin 13.2 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 5e:6c:c:b9:e3:1a
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) .../ ^C

$ sudo /bin/launchctl kickstart -kp system/com.apple.NetworkSharing
service spawned with pid: 5012
$ minikube delete --all
🔥  Deleting "minikube" in qemu2 ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles

$ sudo /bin/launchctl kickstart -kp system/com.apple.NetworkSharing
service spawned with pid: 5134
$ sudo /bin/launchctl kickstart -kp system/io.github.lima-vm.socket_vmnet
service spawned with pid: 5203
$ sudo /bin/launchctl kickstart -kp system/com.apple.NetworkSharing
service spawned with pid: 5239

$ minikube start --driver=qemu2
😄  minikube v1.29.0 on Darwin 13.2 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...- OUTPUT: 
ERROR: Failed to connect to "/var/run/socket_vmnet": Permission denied

🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: Failed to connect to "/var/run/socket_vmnet": Permission denied: exit status 1
^C

$ minikube delete --all
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles

$ sudo chown root:primarygroup /var/run/socket_vmnet

$ minikube start --driver=qemu2
😄  minikube v1.29.0 on Darwin 13.2 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 26:9e:e3:4:84:75
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) .../

medyagh commented 1 year ago

@AkihiroSuda I wonder if could this be related to Crop VPNs ? is there a way to add logs or detect if it is being blocked by a corp VPN?

onatm commented 1 year ago

It seems like this problem comes back after a few days of working without any issue. At first, rebooting and re-installing from the source was working but now it's not.

medyagh commented 1 year ago

It seems like this problem comes back after a few days of working without any issue. At first, rebooting and re-installing from the source was working but now it's not.

@onatm I am curious are you on a Corp Network or behind a VPN ?

mprimeaux commented 1 year ago

@medyagh FWIW, I am not and I also experience the same issue at times and, like @onatm, after a few days it fails. As another interesting data point, this occurs when on my Mac Studio (desktop) and MacBook Pro (laptop). I've played around with power options (e.g. don't sleep and wake on network activity) but the problem still persists.

Fortunately, I've been able to recover using various techniques ranging from a reboot to uninstalling / installing socket_vmnet.

medyagh commented 1 year ago

@medyagh FWIW, I am not and I also experience the same issue at times and, like @onatm, after a few days it fails. As another interesting data point, this occurs when on my Mac Studio (desktop) and MacBook Pro (laptop). I've played around with power options (e.g. don't sleep and wake on network activity) but the problem still persists.

Fortunately, I've been able to recover using various techniques ranging from a reboot to uninstalling / installing socket_vmnet.

for me I haven't recovered yet even after uninstalling/rebooting. @mprimeaux im am curious are u using this with Lima or minikube ? and are you behind a VPN or use it a corp laptop ?

mprimeaux commented 1 year ago

@medyagh I use socket_vmnet for both Lima and Minikube and I run them concurrently. We build our multi-arch container images on Lima and then deploy to Minikube. No corporate VPN involved as the computers aren't corporate assets.

In many cases, what works for me is (1) stop socket_vmnet, (2) uninstall socket_vmnet, (3) reboot, (4) reinstall socket_vmnet, and (5) restart socket_vmnet. All of this is a pain to say the least.

I've been playing around with running k8s in Lima since that allows me to obtain a routable address on my local network (bridged mode) as I have other IoT devices that I'm working with against the k8s cluster. That seems fine.

Also, as another data point, Lima locks up frequently when building our multi-arch container images; but only on macOS ARM64. macOS Intel is stable. The only way to recover is to force a stop limactl stop --force and then rebuild the VM. Again, less than ideal. I haven't had time to reason through hypotheses to either prove or disprove them in search of root cause.

Let me know if I can provide any other details. Just trying to get back above water so I can reconvene on testing various issues and configurations across socket_vmnet, minikube, and lima.

mprimeaux commented 1 year ago

...and thanks for Lima v0.15.0, its much easier to start k8s in Lima with bridged networking enabled.

limactl start --set='.cpus = 4 | .memory = "16GiB" | .networks[0] = { "socket": "/opt/homebrew/var/run/socket_vmnet.bridged.en0"}' template://k8s

Possible before but I had to download the k8s template locally and modify it. This shortens my debug cycles with various configurations.

ShockleyJE commented 1 year ago

Adding confirmation that +1 is experiencing the issue on Ventura 13.0 M1 Max with latest minikube, qemu, and socket_vmnet.

Minikube is fine with only qemu until needing to expose external IP's to test services with external IP's.

I installed socket_vmnet & started the service like such:

brew install socket_vmnet
brew tap homebrew/services
HOMEBREW=$(which brew) && sudo ${HOMEBREW} services start socket_vmnet

I should also note that I was on an ethernet connection & no corp proxy.

If there's anything I can do to help with troubleshooting for y'all please feel free to request.

jandubois commented 1 year ago

Adding confirmation that +1 is experiencing the issue

I'm not sure what "the issue" is in this discussion anymore. As I explained above, the DHCP lease information will never be found on the local machine because it comes from the local network and not the builtin DHCP. You have to determine that IP address from inside the VM, e.g. by also having a shared or host-only interface, whose IP address you can get from the local leases file.

I don't think there is anything left to discuss about that.

If this discussion is about socket_vmnet hanging, then I think there should be a separate Github issue for that, and this issue should be closed.

mprimeaux commented 1 year ago

@jandubois I agree that a different issue should be opened. Sorry for the tangent.

Speaking for my desire (to have Minikube + QEMU driver work in bridged mode), a minikube enhancement request might be the best course of action.

medyagh commented 1 year ago

Adding confirmation that +1 is experiencing the issue> I'm not sure what "the issue" is in this discussion anymore. As I explained above, the DHCP lease information will never be found on the local machine because it comes from the local network and not the builtin DHCP. You have to determine that IP address from inside the VM, e.g. by also having a shared or host-only interface, whose IP address you can get from the local leases file.

@jandubois are you saying even at the times that socket_vmnet with minikube works well with minikube, even at those times the DHCP were not found in minikube either ? I dont think thats has been the case in my experience. when it works, minikube finds the lease on before the 6th try ... but when it doesnt work it wont find it and times out

jandubois commented 1 year ago

I'm sorry, I was only thinking about /var/db/dhcpd_leases in my previous answers, and that file cannot contain entries for bridged network adapters (see the image at the top of the thread). It (dhcpd) only records the leases it has issued itself, not the ones that have been received.

But I just realized that macOS also stores client leases under /var/db/dhcpclient/leases, and it would make sense if it also recorded the leases it has requested for bridged vmnet networks in there. Alas on the machines I just checked this doesn't seem to be the case; I only see leases for host adapters.

afbjorklund commented 1 year ago

In all, querying the IP address via a local slirp connection seems to be the way to do this.

This is the traditional (i.e. VirtualBox) approach, later copied to other drivers such as libvirt

https://github.com/kubernetes/minikube/issues/4938

1) https://github.com/docker/machine/blob/master/drivers/virtualbox/network.go

2) https://github.com/dhiltgen/docker-machine-kvm/blob/master/kvm.go

aelsnz commented 1 year ago

Question here and maybe this is not related, but based on tests I have been running I think this might be - and indicating a change or issue with bootp and the allocation of DHCP addresses to the bridge network.

See - https://github.com/lima-vm/lima/issues/1259

The issue I have seen is that on Mac OS 13.3 M1 is that when I try to use Colima / Lima / Minikube with sock_vmnet and qemu is that the DHCP address is not being allocated.

If I stop bootp, and add basic DHCP server with isc-dhcp I can see that DHCP addresses are being requested via the sock_vmnet/bridge into the VMs.

Now with Minikube it seem to look for specific format of the dhcpd_leases and it must be /var/db/dhcpd_leases so I still get failure on starting minikube but I think this now only due to the format of the leases file being different than expected.

$ minikube start --driver qemu --network socket_vmnet
😄  minikube v1.30.1 on Darwin 13.3.1 (arm64)
✨  Using the qemu2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating qemu2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: invalid line in dhcp leases file: # The format of this file is documented in the dhcpd.leases(5) manual page.
🔥  Creating qemu2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
😿  Failed to start qemu2 VM. Running "minikube delete" may fix it: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: invalid line in dhcp leases file: # The format of this file is documented in the dhcpd.leases(5) manual page.

Basic setup of the DHCP:

sudo /opt/homebrew/opt/isc-dhcp/sbin/dhcpd -d -lf /var/db/dhcpd_leases -cf ./dhcpd-minikube.conf bridge100

Example output I see which shows while minikube is trying to start that it is getting to the DHCP server:

$ sudo /opt/homebrew/opt/isc-dhcp/sbin/dhcpd -d -lf /var/db/dhcpd_leases -cf ./dhcpd-minikube.conf bridge100
Password:
Internet Systems Consortium DHCP Server 4.4.3-P1
Copyright 2004-2022 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Config file: ./dhcpd-minikube.conf
Database file: /var/db/dhcpd_leases
PID file: /opt/homebrew/var/dhcpd/dhcpd.pid
Wrote 0 leases to leases file.
Listening on BPF/bridge100/be:d0:74:fa:8c:64/192.168.105.0/24
Sending on   BPF/bridge100/be:d0:74:fa:8c:64/192.168.105.0/24
Sending on   Socket/fallback/fallback-net
Server starting service.
DHCPDISCOVER from b2:c4:a7:fb:0e:cc via bridge100
DHCPOFFER on 192.168.105.2 to b2:c4:a7:fb:0e:cc (minikube) via bridge100
DHCPREQUEST for 192.168.105.2 (192.168.105.1) from b2:c4:a7:fb:0e:cc (minikube) via bridge100
DHCPACK on 192.168.105.2 to b2:c4:a7:fb:0e:cc (minikube) via bridge100
DHCPDISCOVER from 8a:84:6a:41:30:36 via bridge100
DHCPOFFER on 192.168.105.3 to 8a:84:6a:41:30:36 (minikube) via bridge100
DHCPREQUEST for 192.168.105.3 (192.168.105.1) from 8a:84:6a:41:30:36 (minikube) via bridge100
DHCPACK on 192.168.105.3 to 8a:84:6a:41:30:36 (minikube) via bridge100
...

/opt/homebrew/etc/dhcpd-minikube.conf contents

# dhcpd.conf
default-lease-time 600;
max-lease-time 7200;
authoritative;

subnet 192.168.105.0 netmask 255.255.255.0 {
  range 192.168.105.2 192.168.105.200;
  option domain-name-servers 1.1.1.1, 8.8.8.8;
  option routers 192.168.105.1;
  option broadcast-address 192.168.105.255;

  default-lease-time 6000;
  max-lease-time 72000;
}

AkihiroSuda commented 1 year ago

This might work?

@ aelsnz Finally this worked for me, I have run the below command every time I boot, looks like bootp is being blocked by firewall.
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

Originally posted by @ AravindGopala in https://github.com/lima-vm/lima/issues/1259#issuecomment-1573051614

spowelljr commented 1 year ago

This might work?
@ aelsnz Finally this worked for me, I have run the below command every time I boot, looks like bootp is being blocked by firewall.
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Originally posted by @ AravindGopala in lima-vm/lima#1259 (comment)

I tried on my personal machine and GitHub Actions and both returned The application is not part of the firewall

mprimeaux commented 1 year ago

I also can confirm the same experience as @spowelljr on my macOS ARM machines.

❯ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd 
The application is not part of the firewall

Same message whether my firewall is enabled or not.

jandubois commented 1 year ago

Did you add it to the firewall first?

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Password:
The application is not part of the firewall

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
Application at path ( /usr/libexec/bootpd ) added to firewall

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Incoming connection to the application is permitted

mprimeaux commented 1 year ago

Thanks, @jandubois. I did not but do it now and test.

spowelljr commented 1 year ago

@jandubois That worked for my personal machine, trying on GitHub Actions now, thanks.

mprimeaux commented 1 year ago

@jandubois This worked for my machines, also. Thanks!

medyagh commented 1 year ago

Did you add it to the firewall first?

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Password:
The application is not part of the firewall

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
Application at path ( /usr/libexec/bootpd ) added to firewall

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
Incoming connection to the application is permitted

Thank you so much for finding this out ! this helps a lot of us, this seems to be working on my mac M1 too,

btw this does not on github action, do you have any clue/guess how github action MacOs could be different ?

medyagh commented 1 year ago

This issue is solved on physical macbooks, but still persists on github action macos

I suggest we rename this issue, and it might be helpful to investigate with Github Action Environments to see if they know if any special quirks of gh action envs ?

tmoschou commented 12 months ago

Unfortunately, even with sudo, we cannot run the above workaround commands on Macs enrolled in Mobile Device Management (MDM)

$ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
Password:
Firewall settings cannot be modified from command line on managed Mac computers.

I imagine this would be a common occurrence for developers working for a medium-large company and where difficult to persuade their company's IT department to allow firewall exceptions, though based on my limited knowledges seems doable - https://developer.apple.com/documentation/devicemanagement/firewall/applicationsitem

Is it possible this is actually a bug on Apple's end and if so has anyone reported it https://developer.apple.com/bug-reporting/? We only see this bug only after rebooting where socket_vmnet was enabled on some previous boot via sudo brew services start. Apple's documentation themselves seem to indicate that DHCP shouldn't be blocked (and in instances where we start the service from fresh install isn't)

From https://developer.apple.com/documentation/vmnet#1681413

The interface receives a private IPv4 address via DHCP

I have verified using wireshark that QEMU VM starts/boots and sends through a DHCP Discover request packets (broadcast) port 67 on the vmenet0 and bridge100 interfaces created by socket_vmnet. launchd is listening on UDP port 67 on all interfaces

$ sudo lsof -iUDP:67 -P
Password:
COMMAND PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
launchd   1 root   24u  IPv4 0x2727b31c239489bb      0t0  UDP *:67
launchd   1 root   25u  IPv4 0x2727b31c239489bb      0t0  UDP *:67

Presumably launchd is just not socket activating /System/Library/LaunchDaemons/bootps.plist to fulfil the DHCP request.

dijitali commented 11 months ago

I imagine this would be a common occurrence for developers working for a medium-large company and where difficult to persuade their company's IT department to allow firewall exceptions, though based on my limited knowledges seems doable - https://developer.apple.com/documentation/devicemanagement/firewall/applicationsitem

@tmoschou - unfortunately I think the firewall MDM profile attributes only allows specifying apps via a BundleID and I don't think /usr/libexec/bootpd has a BundleID.

Microsoft InTune docs for creating macOS details a way to get the BundleID of an app but it returns an empty value for bootpd:

~ osascript -e 'id of app "/usr/libexec/bootpd"'

Compared to other things:

~ osascript -e 'id of app "Notes"'
com.apple.Notes

~ osascript -e 'id of app "/usr/libexec/not_a_real_thing"'
0:2: execution error: Can’t get application "/usr/libexec/not_a_real_thing". (-1728)

tmoschou commented 11 months ago

I don't think /usr/libexec/bootpd has a BundleID

Ah maybe com.apple.bootpd might work? Its the Label value used in /System/Library/LaunchDaemons/bootps.plist

avoidik commented 1 month ago

in my case I do not have bootpd entry on the Firewall list UI, but if I decode the com.apple.alf.plist I can see it there anyway

    <key>exceptions</key>
    <array>
        <dict>
            <key>path</key>
            <string>/usr/libexec/bootpd</string>
            <key>state</key>
            <integer>3</integer>
        </dict>
        ...
    </array>

I assume there's no need to enlist it, it would be great if someone could confirm this

lima-vm / socket_vmnet

DHCP leases file not getting updated #18