search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.46k stars 4.88k forks source link

Start minikube on Chromebook Linux (Debian 12) failed #18026

Closed xn01302 closed 4 months ago

xn01302 commented 9 months ago

What Happened?

Hi all.

start minikube on Chromebook Linux (Debian 12) with kvm2 driver failed. It's a problem with /dev/random.

mdersch@penguin:~$ minikube start --driver=kvm2 😄 minikube v1.32.0 on Debian 12.4 (amd64) ✨ Using the kvm2 driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🔄 Restarting existing kvm2 VM for "minikube" ... 🤦 StartHost failed, but will try again: driver start: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted') 🔄 Restarting existing kvm2 VM for "minikube" ... 😿 Failed to start kvm2 VM. Running "minikube delete" may fix it: driver start: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

❌ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: driver start: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

Attach the log file

logs.txt

Operating System

Other

Driver

KVM2

afbjorklund commented 9 months ago

You need some special settings, for running libvirt on ChromeOS

xn01302 commented 9 months ago

Yes, this settings worked with Debian11. Now I have Debian12 and this strange /dev/random problem occurs. I have done all these setting but no success.

xn01302 commented 9 months ago

I'm switched back to Debian 11 (Bullseye). Everything works fine now. It's a problem with Debian 12. I don't know what exactly. The behavior is different.

xn01302 commented 9 months ago

Please let me know if anybody has success with minikube driver=kvm2 and Debian12.

caerulescens commented 9 months ago

@xn01302 I will confirm from experience that this is not an issue with Debian. The kvm2 driver for minikube works great on Debian, and I use it everyday. I know the exact steps to configure KVM for minikube, and I performed these steps recently when I upgraded to Bookworm from Bullseye. You only need to follow first few steps in this guide after performing a vanilla install of Debian (see below).

  1. Install Debian
  2. (any other dependencies)
  3. sudo apt install qemu-system libvirt-daemon-system
  4. sudo adduser $USER libvirt
  5. Logout, then login to update user groups
  6. minikube start --driver=kvm2

Also, try deleting ~/.minikube and then try to start again; minikube will recreate ~/.minikube and download the kvm2 driver.

virsh version:

Compiled against library: libvirt 9.0.0
Using library: libvirt 9.0.0
Using API: QEMU 9.0.0
Running hypervisor: QEMU 7.2.7

minikube version:

minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

lsb_release -a:

Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:    12
Codename:   bookworm

minikube start --driver=kvm2

😄  minikube v1.32.0 on Debian 12.4
✨  Using the kvm2 driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating kvm2 VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
caerulescens commented 9 months ago

@xn01302 You should install kvm like I said, and try: https://github.com/kubernetes/minikube/issues/14272#issuecomment-1154840378

caerulescens commented 9 months ago

/assign

xn01302 commented 9 months ago

Okay, I will try this. Currently I'm working with Debian 11.

caerulescens commented 9 months ago

@xn01302 I can verify that the instructions work on Debian 11 too, so if you have issues, then it could be something else.

caerulescens commented 9 months ago

I think that @afbjorklund is correct because Chrome books end up having quite a few quirks with their security; although, I do not have much experience with running Debian on Chrome books.

afbjorklund commented 9 months ago

libvirt has a command virt-host-validate qemu, which you can use to troubleshoot. The kvm/kvm2 driver uses libvirt.

Alternatively you can try with the qemu/qemu2 driver, which also supports kvm but does not use the libvirtd daemon.

caerulescens commented 9 months ago

@xn01302 Here's what a working configuration will look like using virt-host-validate qemu.

  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : WARN (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (AMD Secure Encrypted Virtualization appears to be disabled in firmware.)

Using virt-host-validate without arguments will yield checking the Linux containers (LXC) configuration as well. 

  QEMU: Checking for hardware virtualization                                 : PASS
  QEMU: Checking if device /dev/kvm exists                                   : PASS
  QEMU: Checking if device /dev/kvm is accessible                            : PASS
  QEMU: Checking if device /dev/vhost-net exists                             : PASS
  QEMU: Checking if device /dev/net/tun exists                               : PASS
  QEMU: Checking for cgroup 'cpu' controller support                         : PASS
  QEMU: Checking for cgroup 'cpuacct' controller support                     : PASS
  QEMU: Checking for cgroup 'cpuset' controller support                      : PASS
  QEMU: Checking for cgroup 'memory' controller support                      : PASS
  QEMU: Checking for cgroup 'devices' controller support                     : WARN (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
  QEMU: Checking for cgroup 'blkio' controller support                       : PASS
  QEMU: Checking for device assignment IOMMU support                         : PASS
  QEMU: Checking if IOMMU is enabled by kernel                               : PASS
  QEMU: Checking for secure guest support                                    : WARN (AMD Secure Encrypted Virtualization appears to be disabled in firmware.)
   LXC: Checking for Linux >= 2.6.26                                         : PASS
   LXC: Checking for namespace ipc                                           : PASS
   LXC: Checking for namespace mnt                                           : PASS
   LXC: Checking for namespace pid                                           : PASS
   LXC: Checking for namespace uts                                           : PASS
   LXC: Checking for namespace net                                           : PASS
   LXC: Checking for namespace user                                          : PASS
   LXC: Checking for cgroup 'cpu' controller support                         : PASS
   LXC: Checking for cgroup 'cpuacct' controller support                     : PASS
   LXC: Checking for cgroup 'cpuset' controller support                      : PASS
   LXC: Checking for cgroup 'memory' controller support                      : PASS
   LXC: Checking for cgroup 'devices' controller support                     : FAIL (Enable 'devices' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'freezer' controller support                     : FAIL (Enable 'freezer' in kernel Kconfig file or mount/enable cgroup controller in your system)
   LXC: Checking for cgroup 'blkio' controller support                       : PASS
   LXC: Checking if device /sys/fs/fuse/connections exists                   : PASS
xn01302 commented 9 months ago

@caerulescens Sorry for delayed response. I'll try it. Have installed a additional container with debian 12 now. Give you feedback in a few days.

vveisard commented 9 months ago

I am having the same issue on my Debian 12 install on ChromeOS. I've already tried editing /etc/libvirt/qemu.conf with the suggested settings

EDIT: I tried the suggested process on a clean Debian 11 install and it worked.

caerulescens commented 9 months ago

I don't have a Chromebook to debug this fix.

xn01302 commented 9 months ago

I've tried again on my chromebook. I have created a new Debian12 container and installed minikube. Start process failed again with the same error message as before. Problem with /dev/random

minikube start --driver=kvm2 😄 minikube v1.32.0 on Debian 12.5 (amd64) ✨ Using the kvm2 driver based on user configuration 💾 Downloading driver docker-machine-driver-kvm2:

docker-machine-driver-kvm2-...: 65 B / 65 B [---------] 100.00% ? p/s 0s docker-machine-driver-kvm2-...: 13.01 MiB / 13.01 MiB 100.00% 10.07 MiB 💿 Downloading VM boot image ... minikube-v1.32.1-amd64.iso....: 65 B / 65 B [---------] 100.00% ? p/s 0s minikube-v1.32.1-amd64.iso: 292.96 MiB / 292.96 MiB 100.00% 7.50 MiB p/ 👍 Starting control plane node minikube in cluster minikube 💾 Downloading Kubernetes v1.28.3 preload ... preloaded-images-k8s-v18-v1...: 403.35 MiB / 403.35 MiB 100.00% 7.76 Mi 🔥 Creating kvm2 VM (CPUs=2, Memory=2200MB, Disk=20000MB) ... 🔥 Deleting "minikube" in kvm2 ... 🤦 StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted') 🔥 Creating kvm2 VM (CPUs=2, Memory=2200MB, Disk=20000MB) ... 😿 Failed to start kvm2 VM. Running "minikube delete" may fix it: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

❌ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: Error creating machine: Error in driver during machine creation: error creating VM: virError(Code=38, Domain=10, Message='Failed to chown device /dev/random: Operation not permitted')

afbjorklund commented 9 months ago

If it is a blocker, it is possibly to make the RNG block in the virt configuration more optional...

    <rng model='virtio'>
      <backend model='random'>/dev/random</backend>
    </rng>

There are other blocks (such as GPU), that are optional and toggled with flags at runtime.

norseto commented 8 months ago

I looked at https://gitlab.com/libvirt/libvirt/-/issues/556 and set namespaces=[] in /etc/libvirt/qemu.conf and was able to start minikube. I hope this is helpful.

xn01302 commented 7 months ago

Hi, thanks a lot. It works with namespaces = []. Great!!!

Best regards Mirko

caerulescens commented 6 months ago

/close

k8s-ci-robot commented 6 months ago

@caerulescens: You can't close an active issue/PR unless you authored it or you are a collaborator.

In response to [this](https://github.com/kubernetes/minikube/issues/18026#issuecomment-2106400881): >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
caerulescens commented 6 months ago

@xn01302 Awesome; close the issue when you can