search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.5k stars 4.89k forks source link

minikube (qemu2) failes to start with "failed to get IP address" #15960

Open denisko opened 1 year ago

denisko commented 1 year ago

What Happened?

Minikube fails to start: Exiting due to GUEST_PROVISION: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for xx:xx:xx:xx:xx:xx

Attach the log file

minikube-logs.txt minikube_logs_e4032c57df89461a76cf8ea248c98ea6558dccb1_0.log

Operating System

macOS (Default)

Driver

QEMU

afbjorklund commented 1 year ago

Some kind of compatibility issue with socket_vmnet, it only lists one DHCP entry coming from "colima"

I0303 10:05:06.492396   25101 main.go:141] libmachine: Searching for 2:a8:48:27:20:f2 in /var/db/dhcpd_leases ...
I0303 10:05:06.492450   25101 main.go:141] libmachine: Found 1 entries in /var/db/dhcpd_leases!
I0303 10:05:06.492474   25101 main.go:141] libmachine: dhcp entry: {Name:colima IPAddress:192.168.106.2 HWAddress:52:55:55:da:34:59 ID:1,52:55:55:da:34:59 Lease:0x63eb6dd7}
mprimeaux commented 1 year ago

I'm not sure what is going on yet but I am also experiencing this issue, which seems to be a regression as compared to versions before minikube 1.30.x but I do need to verify.

I'll post logs but, in summary, I'm unable to successfully start a minikube cluster when using socket_vmnet (via brew or direct source compilation) in NAT or bridged mode. This used to work but does not even after following steps outlined in Known Issues.

Here are my machine details:

❯ macchina                                                                                                                                                 ─╯

                  ,MMMM.           Host        -  <REMOVED>
                .MMMMMM            Machine     -  Mac13,2
                MMMMM,             Kernel      -  22.4.0
      .;MMMMM:' MMMMMMMMMM;.       OS          -  macOS 13.3.1 Ventura
    MMMMMMMMMMMMNWMMMMMMMMMMM:     DE          -  Aqua
  .MMMMMMMMMMMMMMMMMMMMMMMMWM.     WM          -  Quartz Compositor
  MMMMMMMMMMMMMMMMMMMMMMMMM.       Packages    -  263 (Homebrew)
 ;MMMMMMMMMMMMMMMMMMMMMMMM:        Terminal    -  iTerm2 (Version 3.4.19)
 :MMMMMMMMMMMMMMMMMMMMMMMM:        Shell       -  zsh
 .MMMMMMMMMMMMMMMMMMMMMMMMM.       Uptime      -  3d 5h 49m
  MMMMMMMMMMMMMMMMMMMMMMMMMMM.     CPU         -  Apple M1 Ultra (20)
   .MMMMMMMMMMMMMMMMMMMMMMMMMM.    Resolution  -  5120x2880@60fps (as 2560x1440)
     MMMMMMMMMMMMMMMMMMMMMMMM                     5120x2880@60fps (as 2560x1440)
      ;MMMMMMMMMMMMMMMMMMMM.       Brightness  -  100%
        .MMMM,.    .MMMM,.         CPU Load    -  18%
                                   Memory      -  60.2 GB/134.2 GB

Minikube version is:

minikube version: v1.30.1
commit: 08896fd1dc362c097c925146c4a0d0dac715ace0

I will do more testing tomorrow.

mprimeaux commented 1 year ago

FWIW, here are my logs from today's latest attempt to start minikube with socket_vmnet.

minikube_logs_246a8c5e046f473e09b808593858b4737e8d9296_0.log

❯ minikube config view
- container-runtime: containerd
- cpus: 6
- driver: qemu2
- memory: 16384
❯ minikube start --network socket_vmnet
😄  minikube v1.30.1 on Darwin 13.3.1 (arm64)
✨  Using the qemu2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.26.3 preload ...
    > preloaded-images-k8s-v18-v1...:  358.98 MiB / 358.98 MiB  100.00% 44.94 M
🔥  Creating qemu2 VM (CPUs=6, Memory=16384MB, Disk=20000MB) ...
🔥  Deleting "minikube" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 5e:40:c0:5c:ab:86
🔥  Creating qemu2 VM (CPUs=6, Memory=16384MB, Disk=20000MB) ...
😿  Failed to start qemu2 VM. Running "minikube delete" may fix it: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 42:74:86:6:b8:92

❌  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: IP address never found in dhcp leases file: failed to get IP address: could not find an IP address for 42:74:86:6:b8:92

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

I'm using socket_vmnet via brew but not in bridged mode. 

❯ sudo brew services info socket_vmnet 
socket_vmnet (homebrew.mxcl.socket_vmnet)
Running: ✔
Loaded: ✔
Schedulable: ✘
User: root
PID: 743

I've also tried different values of container-runtime to no avail. I'll find more time tomorrow to debug a bit deeper.

torenware commented 1 year ago

Just tried this with newly released minikube v1.31.0. Still busted on amd64 (might be working as 1.30.x on arm64!)

mprimeaux commented 1 year ago

Still busted on arm64, also. As several folks have mentioned, the fix for me on both amd64 and arm64 is to restart socket_vmnet via brew services and then to run the following:

minikube delete
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

This workaround works for me 100% of the time.

spowelljr commented 1 year ago

minikube v1.31.0 should unblock the firewall if it detects it's being blocked.

$ $ ./out/minikube start --driver qemu
😄  minikube v1.31.0 on Darwin 13.4.1 (arm64)
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
🔑  Your firewall is blocking bootpd which is required for socket_vmnet. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd 
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd 

Password: 
👍  Starting control plane node minikube in cluster minikube
💾  Downloading Kubernetes v1.27.3 preload ...
    > preloaded-images-k8s-v18-v1...:  327.72 MiB / 327.72 MiB  100.00% 50.68 M
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
❗  This VM is having trouble accessing https://registry.k8s.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.27.3 on Docker 24.0.4 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

But restarting socket_vmnet via brew services is also required to get it working for you @mprimeaux?

torenware commented 1 year ago

Still busted on arm64, also. As several folks have mentioned, the fix for me on both amd64 and arm64 is to restart socket_vmnet via brew services and then to run the following:

minikube delete
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

This workaround works for me 100% of the time.

I can't get it to run on amd64, even with the ownership on /usr/local/var/run/socket_vmnet set to root:admin (which makes it possible to run minikube start as non-root). Here's what I see on my Intel Macbook Pro, immediately after giving the user password for sudo:

Robs-MBP:images myuser$ minikube -p vmnet delete
🔥  Deleting "vmnet" in qemu2 ...
💀  Removed all traces of the "vmnet" cluster.
Robs-MBP:images myuser$ minikube -p vmnet start --driver qemu
😄  [vmnet] minikube v1.31.0 on Darwin 11.7.8
✨  Using the qemu2 driver based on user configuration
🌐  Automatically selected the socket_vmnet network
👍  Starting control plane node vmnet in cluster vmnet
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for socket_vmnet. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

🔄  Successfully unblocked bootpd process from firewall, retrying
🔥  Deleting "vmnet" in qemu2 ...
🤦  StartHost failed, but will try again: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for 32:82:30:a9:ad:e1
🔥  Creating qemu2 VM (CPUs=2, Memory=4000MB, Disk=20000MB) ...
🔑  Your firewall is blocking bootpd which is required for socket_vmnet. The following commands will be executed to unblock bootpd:

    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
    $ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

🔄  Successfully unblocked bootpd process from firewall, retrying
😿  Failed to start qemu2 VM. Running "minikube delete -p vmnet" may fix it: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for b2:f9:59:88:37:d6

❌  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for b2:f9:59:88:37:d6

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│                                                                                           │
│    😿  If the above advice does not help, please let us know:                             │
│    👉  https://github.com/kubernetes/minikube/issues/new/choose                           │
│                                                                                           │
│    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    │
│                                                                                           │
╰───────────────────────────────────────────────────────────────────────────────────────────╯
mprimeaux commented 1 year ago

Did you run sudo brew services restart socket_vmnet prior to the above?

torenware commented 1 year ago

@mprimeaux Yep. Here's the status:

Robs-MBP:images myuser$ sudo brew services info socket_vmnet
socket_vmnet (homebrew.mxcl.socket_vmnet)
Running: ✔
Loaded: ✔
Schedulable: ✘
User: root
PID: 94341
mprimeaux commented 1 year ago

@torenware There are days I wish I were on a beach :)

Interestingly enough, I had this very scenario happen when I was in an airport last week. Turns our, the captured network was preventing my VM from obtaining a DHCP address. By chance, are you on a captured network?

torenware commented 1 year ago

File:

@torenware There are days I wish I were on a beach :)

Interestingly enough, I had this very scenario happen when I was in an airport last week. Turns our, the captured network was preventing my VM from obtaining a DHCP address. By chance, are you on a captured network?

Nope. Plain vanilla wifi, no fancy stuff going on.

torenware commented 1 year ago

logs.txt

haroldship commented 1 year ago

I also have this issue, Macbook Pro 16" 2019 Intel running Ventura 13.5. logs.txt minikube_logs_0720879eb88b105f38a2418cc93cdf68b441630a_0.log

richpryce commented 1 year ago

I am having this exact same issue as well. Has a fix been found?

blind3dd commented 1 year ago

I think I have found a workaround + there is a code change needed I've just created.

First you start colima, make sure it works, then socket_vmnet.

Also make sure to define proper firewall rules for bridge created. When colima starts it defines the port, make sure your lease is something with 127.0.0.1 in it matching the mac address found through the leases from /var/db/dhcpd_leases.

What is more, you're going to need to edit the ~/.minikube/machines/<your_machine>/config.json to put there SSHPort and SSHKeyPath of lima ($HOME/.lima/_config/user), I have gone with ssh-add because colima ~/.colima/ssh_config has ForwardAgent yes, anyway the Port there is generated different every time, so probably this PR below should be updated to discover it automatically and read from ssh_config of colima.

After editing a profile for colima/preparing it (containerd, sshfs):

colima start --cpu 4 --memory 8 --runtime containerd --vm-type qemu --ssh-agent --very-verbose --arch aarch64

minikube start --driver qemu2 --network socket_vmnet --cpus 4 --memory 8192 --disk-size 60G --container-runtime containerd --namespace default -p kubernetes-colima --network socket_vmnet --alsologtostderr -v=7

I have also made sure I've added proper anchor and pfctl rules for the bridge created as mentioned above and made sure other firewalls are not blocking bootpd (system and apps).

Basically this needs to appear (in my case) - username cannot be overriden hence the initial PR. ssh -p 49590 usualsuspectx@127.0.0.1 Welcome to Alpine! The Alpine Wiki contains a large amount of how-to guides and general information about administrating Alpine systems. See <https://wiki.alpinelinux.org/>. You can setup the system with the command: setup-alpine You may change this message by editing /etc/motd. colima:~$

λ /Users/ colima list PROFILE STATUS ARCH CPUS MEMORY DISK RUNTIME ADDRESS default Running aarch64 2 4GiB 60GiB containerd

Initial PR: https://github.com/kubernetes/minikube/pull/17602/commits/7e9f3a6a2f7f131d6d1b8a94cd6ffe4472e3dc53

PR has been updated to cover the SSHPort discovery of colima machine and its ssh_config.

afbjorklund commented 1 year ago

It was not intended to start two VMs, so it seems strange to have both colima and minikube qemu2 running.

Overriding the user name is something that could be considered for all drivers, but it is not needed for the kic image.

For colima, it normally starts k3s when you run it with --kubernetes.

colima start --kubernetes

For lima, it simply runs kubeadm on the node instead of using minikube.

limactl start template://k8s
blind3dd commented 1 year ago

Agreed that it is not common, I had to run vmnet separately having those 2 machines and I do not run k3s there with colima, however I was able to obtain the minikube address with the usage of localhost and MAC address in this setup. I am generally thinking about moving this towards lima driver (or dig into qemu2 even deeper) and it's just how I started: see the ip below:


|      Profile      | VM Driver |  Runtime   |    IP     | Port | Version | Status  | Nodes | Active |
|-------------------|-----------|------------|-----------|------|---------|---------|-------|--------|
| kubernetes-colima | qemu2     | containerd | 10.0.2.15 | 8443 | v1.27.4 | Stopped |     1 |        |
|-------------------|-----------|------------|-----------|------|---------|---------|-------|--------|```
blind3dd commented 11 months ago

Agreed that 2 VMs are not intended to start, I changed the course and worked with dhcp leases a little bit, created proper bridge with vmnet and used lima without colima, now it looks correct, getting myself to speed with the project, can't wait to add more nodes

λ ~/ minikube profile list ------------------- ----------- ------------ ------------- ------ --------- --------- ------- -------- Profile VM Driver Runtime IP Port Version Status Nodes Active
kubernetes-colima qemu2 containerd 192.168.5.4 8443 v1.28.4 Running 1
------------------- ----------- ------------ ------------- ------ --------- --------- ------- --------

λ ~/ kubectl get po -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-5dd5756b68-hkgbw 1/1 Running 0 41s kube-system etcd-kubernetes-colima 1/1 Running 0 54s kube-system kube-apiserver-kubernetes-colima 1/1 Running 0 54s kube-system kube-controller-manager-kubernetes-colima 1/1 Running 0 54s kube-system kube-proxy-q4k9b 1/1 Running 0 42s kube-system kube-scheduler-kubernetes-colima 1/1 Running 0 54s kube-system storage-provisioner 1/1 Running 0 54s

λ ~/ k get no NAME STATUS ROLES AGE VERSION kubernetes-colima Ready control-plane 4m1s v1.28.4`

If I am able to find some free time I will see if this issue can be somehow addressed - need to make sure I have full understanding of the setup. Starts getting proper shapes.

Started with: minikube start --driver qemu2 --cpus 4 --memory 8192 --disk-size 60G --container-runtime containerd --namespace default -p kubernetes-colima --alsologtostderr -v=7 --cni=bridge

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

obnoxxx commented 5 months ago

@mprimeaux wrote:

Still busted on arm64, also. As several folks have mentioned, the fix for me on both amd64 and arm64 is to restart socket_vmnet via brew services and then to run the following:

minikube delete
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

This workaround works for me 100% of the time.

This worked for me once yesterday, and I was so happy, but today it fails again:

GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: creating: ip not found: failed to get IP address: could not find an IP address for 66:17:20:72:c8:65

In my case, the firewall does not seem to be the problem because bootpdis listed there. But my suspicion is that the problem is that bootpdis not even running. How can I get it to run?

obnoxxx commented 5 months ago

I managed to get bootpd running by doing sudo /usr/libexec/bootpd but it doesnt help minikube starting. Still failing with failed to get IP address

k8s-triage-robot commented 2 weeks ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale