search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.21k stars 4.87k forks source link

Qemu on windows #15709

Open medyagh opened 1 year ago

medyagh commented 1 year ago

What Happened?

currently we do not return the Qemu Firmware path for windows

https://github.com/kubernetes/minikube/blob/1784105c66c9060eaa447efa3dab6ac2918167ca/pkg/minikube/registry/drvs/qemu2/qemu2.go#L69

I tried to add this if runtime.GOOS == "windows" { return "C:\Program Files\qemu\share\edk2-x86_64-code.fd", nil }

but my current problem is I dont have a succesfull qemu installation on windows, since the exe binary seems to return nothing but blank space

Attach the log file


PS C:\Users\M\Documents\GitHub\minikube> minikube delete --all --purge
🔥  Removing C:\Users\M\.minikube\machines\minikube ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles
💀  Successfully purged minikube directory located at - [C:\Users\M\.minikube]
PS C:\Users\M\Documents\GitHub\minikube> .\out\minikube.exe start --driver=qemu2 --alsologtostderr
I0125 12:45:35.162601   10756 out.go:296] Setting OutFile to fd 104 ...
I0125 12:45:35.163602   10756 out.go:348] isatty.IsTerminal(104) = true
I0125 12:45:35.163602   10756 out.go:309] Setting ErrFile to fd 108...
I0125 12:45:35.163602   10756 out.go:348] isatty.IsTerminal(108) = true
W0125 12:45:35.178708   10756 root.go:311] Error reading config file at C:\Users\M\.minikube\config\config.json: open C:\Users\M\.minikube\config\config.json: The system cannot find the path specified.
I0125 12:45:35.184407   10756 out.go:303] Setting JSON to false
I0125 12:45:35.195447   10756 start.go:125] hostinfo: {"hostname":"DESKTOP-CNIVAJV","uptime":325076,"bootTime":1674354459,"procs":308,"os":"windows","platform":"Microsoft Windows 11 Pro","platformFamily":"Standalone Workstation","platformVersion":"10.0.22621.1105 Build 22621.1105","kernelVersion":"10.0.22621.1105 Build 22621.1105","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"05a6b6f3-ae09-43f7-9085-ff1d48d12628"}
W0125 12:45:35.195979   10756 start.go:133] gopshost.Virtualization returned error: not implemented yet
I0125 12:45:35.212678   10756 out.go:177] 😄  minikube v1.28.0 on Microsoft Windows 11 Pro 10.0.22621.1105 Build 22621.1105
😄  minikube v1.28.0 on Microsoft Windows 11 Pro 10.0.22621.1105 Build 22621.1105
I0125 12:45:35.213863   10756 notify.go:220] Checking for updates...
W0125 12:45:35.213863   10756 preload.go:295] Failed to list preload files: open C:\Users\M\.minikube\cache\preloaded-tarball: The system cannot find the file specified.
I0125 12:45:35.214371   10756 driver.go:365] Setting default libvirt URI to qemu:///system
I0125 12:45:35.221232   10756 out.go:177] ✨  Using the qemu2 driver based on user configuration
✨  Using the qemu2 driver based on user configuration
I0125 12:45:35.222270   10756 start.go:296] selected driver: qemu2
I0125 12:45:35.222270   10756 start.go:857] validating driver "qemu2" against <nil>
I0125 12:45:35.222799   10756 start.go:868] status for qemu2: {Installed:true Healthy:true Running:true NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0125 12:45:35.222799   10756 start_flags.go:305] no existing cluster config was found, will generate one from the flags
I0125 12:45:35.223863   10756 out.go:177] 🌐  Automatically selected the user network
🌐  Automatically selected the user network
W0125 12:45:35.224381   10756 out.go:239] ❗  You are using the QEMU driver without a dedicated network, which doesn't support `minikube service` & `minikube tunnel` commands.
❗  You are using the QEMU driver without a dedicated network, which doesn't support `minikube service` & `minikube tunnel` commands.
I0125 12:45:35.338704   10756 start_flags.go:386] Using suggested 6000MB memory alloc based on sys=32653MB, container=0MB
I0125 12:45:35.338704   10756 start_flags.go:899] Wait components to verify : map[apiserver:true system_pods:true]
I0125 12:45:35.339227   10756 cni.go:84] Creating CNI manager for ""
I0125 12:45:35.339227   10756 cni.go:157] "qemu2" driver + "docker" container runtime found on kubernetes v1.24+, recommending bridge
I0125 12:45:35.339227   10756 start_flags.go:314] Found "bridge CNI" CNI - setting NetworkPlugin=cni
I0125 12:45:35.339227   10756 start_flags.go:319] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.36-1674164627-15541@sha256:0a2280301e955e0d3910d6e639e0b7341db1f4a25558521ac97b38c782c6189a Memory:6000 CPUs:2 DiskSize:20000 VMDriver: Driver:qemu2 HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.26.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network:user Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:C:\Users\M:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath:/opt/socket_vmnet/bin/socket_vmnet_client SocketVMnetPath:/var/run/socket_vmnet StaticIP:}
I0125 12:45:35.340274   10756 iso.go:125] acquiring lock: {Name:mk366b989fd762419235b23a969f23d3c16eeb8f Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0125 12:45:35.342372   10756 out.go:177] 💿  Downloading VM boot image ...
💿  Downloading VM boot image ...
I0125 12:45:35.342909   10756 download.go:101] Downloading: https://storage.googleapis.com/minikube-builds/iso/15541/minikube-v1.28.0-1674164545-15541-amd64.iso?checksum=file:https://storage.googleapis.com/minikube-builds/iso/15541/minikube-v1.28.0-1674164545-15541-amd64.iso.sha256 -> C:\Users\M\.minikube\cache\iso\amd64\minikube-v1.28.0-1674164545-15541-amd64.iso
    > minikube-v1.28.0-1674164545...:  65 B / 65 B [---------] 100.00% ? p/s 0s
    > minikube-v1.28.0-1674164545...:  276.34 MiB / 276.34 MiB  100.00% 61.11 M
I0125 12:45:40.930612   10756 out.go:177] 👍  Starting control plane node minikube in cluster minikube
👍  Starting control plane node minikube in cluster minikube
I0125 12:45:40.934665   10756 preload.go:132] Checking if preload exists for k8s version v1.26.1 and runtime docker
I0125 12:45:40.986238   10756 preload.go:119] Found remote preload: https://storage.googleapis.com/minikube-preloaded-volume-tarballs/v18/v1.26.1/preloaded-images-k8s-v18-v1.26.1-docker-overlay2-amd64.tar.lz4
I0125 12:45:40.986238   10756 cache.go:57] Caching tarball of preloaded images
I0125 12:45:40.986743   10756 preload.go:132] Checking if preload exists for k8s version v1.26.1 and runtime docker
I0125 12:45:40.993082   10756 out.go:177] 💾  Downloading Kubernetes v1.26.1 preload ...
💾  Downloading Kubernetes v1.26.1 preload ...
I0125 12:45:40.994173   10756 preload.go:238] getting checksum for preloaded-images-k8s-v18-v1.26.1-docker-overlay2-amd64.tar.lz4 ...
I0125 12:45:41.070232   10756 download.go:101] Downloading: https://storage.googleapis.com/minikube-preloaded-volume-tarballs/v18/v1.26.1/preloaded-images-k8s-v18-v1.26.1-docker-overlay2-amd64.tar.lz4?checksum=md5:c6cc8ea1da4e19500d6fe35540785ea8 -> C:\Users\M\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v18-v1.26.1-docker-overlay2-amd64.tar.lz4
    > preloaded-images-k8s-v18-v1...:  397.05 MiB / 397.05 MiB  100.00% 56.06 M
I0125 12:45:49.094976   10756 preload.go:249] saving checksum for preloaded-images-k8s-v18-v1.26.1-docker-overlay2-amd64.tar.lz4 ...
I0125 12:45:49.095981   10756 preload.go:256] verifying checksum of C:\Users\M\.minikube\cache\preloaded-tarball\preloaded-images-k8s-v18-v1.26.1-docker-overlay2-amd64.tar.lz4 ...
I0125 12:45:50.406202   10756 cache.go:60] Finished verifying existence of preloaded tar for  v1.26.1 on docker
I0125 12:45:50.406202   10756 profile.go:148] Saving config to C:\Users\M\.minikube\profiles\minikube\config.json ...
I0125 12:45:50.406202   10756 lock.go:35] WriteFile acquiring C:\Users\M\.minikube\profiles\minikube\config.json: {Name:mk0ef3ad5e9e60bcbe14343e75b513d6a838f166 Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
I0125 12:45:50.407203   10756 cache.go:193] Successfully downloaded all kic artifacts
I0125 12:45:50.407203   10756 start.go:364] acquiring machines lock for minikube: {Name:mk2ce3ae932dafc784f3630c04a10793dd917b3f Clock:{} Delay:500ms Timeout:13m0s Cancel:<nil>}
I0125 12:45:50.407203   10756 start.go:368] acquired machines lock for "minikube" in 0s
I0125 12:45:50.407203   10756 start.go:93] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube-builds/iso/15541/minikube-v1.28.0-1674164545-15541-amd64.iso KicBaseImage:gcr.io/k8s-minikube/kicbase-builds:v0.0.36-1674164627-15541@sha256:0a2280301e955e0d3910d6e639e0b7341db1f4a25558521ac97b38c782c6189a Memory:6000 CPUs:2 DiskSize:20000 VMDriver: Driver:qemu2 HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:0 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.26.1 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.26.1 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network:user Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:C:\Users\M:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath:/opt/socket_vmnet/bin/socket_vmnet_client SocketVMnetPath:/var/run/socket_vmnet StaticIP:} &{Name: IP: Port:8443 KubernetesVersion:v1.26.1 ContainerRuntime:docker ControlPlane:true Worker:true}
I0125 12:45:50.408210   10756 start.go:125] createHost starting for "" (driver="qemu2")
I0125 12:45:50.421829   10756 out.go:204] 🔥  Creating qemu2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🔥  Creating qemu2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...| I0125 12:45:50.423058   10756 start.go:159] libmachine.API.Create for "minikube" (driver="qemu2")
I0125 12:45:50.423058   10756 client.go:168] LocalClient.Create starting
I0125 12:45:50.423594   10756 main.go:141] libmachine: Creating CA: C:\Users\M\.minikube\certs\ca.pem
I0125 12:45:50.492170   10756 main.go:141] libmachine: Creating client certificate: C:\Users\M\.minikube\certs\cert.pem
/ I0125 12:45:50.549644   10756 main.go:141] libmachine: port range: 0 -> 65535
I0125 12:45:50.550642   10756 main.go:141] libmachine: Downloading C:\Users\M\.minikube\cache\boot2docker.iso from file://C:/Users/M/.minikube/cache/iso/amd64/minikube-v1.28.0-1674164545-15541-amd64.iso...
| I0125 12:45:51.300485   10756 main.go:141] libmachine: Creating SSH key...
/ I0125 12:45:51.417059   10756 main.go:141] libmachine: Creating Disk image...
I0125 12:45:51.417059   10756 main.go:141] libmachine: Creating 20000 MB hard disk image...
I0125 12:45:51.433983   10756 main.go:141] libmachine: executing: qemu-img convert -f raw -O qcow2 C:\Users\M\.minikube\machines\minikube\disk.qcow2.raw C:\Users\M\.minikube\machines\minikube\disk.qcow2
- I0125 12:45:51.501254   10756 main.go:141] libmachine: STDOUT:
I0125 12:45:51.501395   10756 main.go:141] libmachine: STDERR:
I0125 12:45:51.507168   10756 main.go:141] libmachine: executing: qemu-img resize C:\Users\M\.minikube\machines\minikube\disk.qcow2 +20000M
I0125 12:45:51.532533   10756 main.go:141] libmachine: STDOUT: Image resized.

I0125 12:45:51.532533   10756 main.go:141] libmachine: STDERR:
I0125 12:45:51.532533   10756 main.go:141] libmachine: DONE writing to C:\Users\M\.minikube\machines\minikube\disk.qcow2.raw and C:\Users\M\.minikube\machines\minikube\disk.qcow2
I0125 12:45:51.533042   10756 main.go:141] libmachine: Starting QEMU VM...
I0125 12:45:51.538805   10756 main.go:141] libmachine: executing: qemu-system-x86_64 -display none -m 6000 -smp 2 -boot d -cdrom C:\Users\M\.minikube\machines\minikube\boot2docker.iso -qmp unix:C:\Users\M\.minikube\machines\minikube\monitor,server,nowait -pidfile C:\Users\M\.minikube\machines\minikube\qemu.pid -nic user,model=virtio,hostfwd=tcp::52476-:22,hostfwd=tcp::52477-:2376,hostname=minikube -daemonize C:\Users\M\.minikube\machines\minikube\disk.qcow2
I0125 12:45:51.562310   10756 main.go:141] libmachine: STDOUT: 
I0125 12:45:51.562310   10756 main.go:141] libmachine: STDERR:
OUTPUT:
ERROR:
I0125 12:45:51.562815   10756 client.go:171] LocalClient.Create took 1.1397571s
/ I0125 12:45:53.576702   10756 start.go:128] duration metric: createHost completed in 3.1684925s
I0125 12:45:53.576702   10756 start.go:83] releasing machines lock for "minikube", held for 3.1694994s
W0125 12:45:53.577236   10756 start.go:678] error starting host: creating host: create: creating: exit status 0xc0000139
I0125 12:45:53.579357   10756 out.go:177] 🔥  Deleting "minikube" in qemu2 ...

🔥  Deleting "minikube" in qemu2 ...
W0125 12:45:53.601459   10756 start.go:683] delete host: api remove: remove C:\Users\M\.minikube\machines\minikube\id_rsa: The process cannot access the file because it is being used by another process.
W0125 12:45:53.601459   10756 out.go:239] 🤦  StartHost failed, but will try again: creating host: create: creating: exit status 0xc0000139
🤦  StartHost failed, but will try again: creating host: create: creating: exit status 0xc0000139
I0125 12:45:53.601991   10756 start.go:693] Will try again in 5 seconds ...
I0125 12:45:58.606539   10756 start.go:364] acquiring machines lock for minikube: {Name:mk2ce3ae932dafc784f3630c04a10793dd917b3f Clock:{} Delay:500ms Timeout:13m0s Cancel:<nil>}
I0125 12:45:58.606539   10756 start.go:368] acquired machines lock for "minikube" in 0s
I0125 12:45:58.607045   10756 start.go:96] Skipping create...Using existing machine configuration
I0125 12:45:58.607045   10756 fix.go:55] fixHost starting:
I0125 12:45:58.607045   10756 fix.go:57] fixHost completed within 0s
I0125 12:45:58.607045   10756 start.go:83] releasing machines lock for "minikube", held for 505.4µs
W0125 12:45:58.607571   10756 out.go:239] 😿  Failed to start qemu2 VM. Running "minikube delete" may fix it: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\M\.minikube\machines\minikube\config.json: The system cannot find the file specified.
😿  Failed to start qemu2 VM. Running "minikube delete" may fix it: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\M\.minikube\machines\minikube\config.json: The system cannot find the file specified.
I0125 12:45:58.619158   10756 out.go:177]

W0125 12:45:58.620224   10756 out.go:239] ❌  Exiting due to GUEST_NOT_FOUND: Failed to start host: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\M\.minikube\machines\minikube\config.json: The system cannot find the file specified.
❌  Exiting due to GUEST_NOT_FOUND: Failed to start host: error loading existing host. Please try running [minikube delete], then run [minikube start] again: filestore "minikube": open C:\Users\M\.minikube\machines\minikube\config.json: The system cannot find the file specified.
W0125 12:45:58.620763   10756 out.go:239] 💡  Suggestion: minikube is missing files relating to your guest environment. This can be fixed by running 'minikube delete'
💡  Suggestion: minikube is missing files relating to your guest environment. This can be fixed by running 'minikube delete'
W0125 12:45:58.620763   10756 out.go:239] 🍿  Related issue: https://github.com/kubernetes/minikube/issues/9130
🍿  Related issue: https://github.com/kubernetes/minikube/issues/9130
I0125 12:45:58.622394   10756 out.go:177]

Operating System

None

Driver

None

afbjorklund commented 1 year ago

It seems to be missing the -accel whpx flag, but there are lots of other things needed to run QEMU on Windows.

        // hardware acceleration is important, it increases performance by 10x
        if runtime.GOOS == "darwin" {
                // On macOS, enable the Hypervisor framework accelerator.
                startCmd = append(startCmd,
                        "-accel", "hvf")
        } else if _, err := os.Stat("/dev/kvm"); err == nil && runtime.GOOS == "linux" {
                // On Linux, enable the Kernel Virtual Machine accelerator.
                startCmd = append(startCmd,
                        "-accel", "kvm")
        } else if runtime.GOOS == "windows" {
                // On Windows, enable the WHPX (Hyper-V) accelerator.
                startCmd = append(startCmd,
                        "-accel", "whpx")
        }

But it should be possible to boot it, with the Windows qemu binaries:

https://qemu.weilnetz.de/w64/

afbjorklund commented 1 year ago
spowelljr commented 1 year ago

Experimental Windows support added with https://github.com/kubernetes/minikube/pull/15781