search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.19k stars 4.87k forks source link

Wrong CPU emulation on minikube qemu driver #14607

Closed gilbahat closed 1 year ago

gilbahat commented 2 years ago

What Happened?

For some reason unclear to me, qemu will default to emulated CPU when running under minikube:

➜ ~ minikube ssh (snipped) $ cat /proc/cpuinfo |grep -i model model : 107 model name : QEMU Virtual CPU version 2.5+ model : 107 model name : QEMU Virtual CPU version 2.5+

this is the command line that generated this machine:

➜ ~ ps aux | grep -i qemu gilbahat 12640 83.6 25.0 40808748 4196160 ?? R 8:02PM 133:03.40 qemu-system-x86_64 -display none -accel hvf -m 6144 -smp 2 -boot d -cdrom /Users/gilbahat/.minikube/machines/minikube/boot2docker.iso -qmp unix:/Users/gilbahat/.minikube/machines/minikube/monitor,server,nowait -pidfile /Users/gilbahat/.minikube/machines/minikube/qemu.pid -nic user,model=virtio,hostfwd=tcp::64985-:22,hostfwd=tcp::64986-:2376,hostname=minikube -daemonize /Users/gilbahat/.minikube/machines/minikube/disk.qcow2

qemu docs suggest the default would be to use the host cpu. in particular on my system, this caused disabling of AVX flags which stopped mongo 5 from working, but this can have any other potentially undesirable side effects as host is recommended according to qemu docs.

➜ ~ qemu-system-x86_64 --version QEMU emulator version 7.0.0 Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

perhaps it is better to explicitly pass a -cpu host flag to qemu-system-x86_64?

Attach the log file

log.txt

System

MacBook Pro (15-inch, 2018)

Operating System

macOS (monterey) x86_64

Driver

qemu

afbjorklund commented 2 years ago

perhaps it is better to explicitly pass a -cpu host flag to qemu-system-x86_64?

Sounds reasonable, the only issue seen so far was that it is not available on all OS.

i.e. from the discussion in lima, it seems to be missing on Windows and on BSD

For the libmachine driver, the whole thing was punted to a "cpu" flag for the user...

   --qemu-machine                                           Machine
   --qemu-cpu                                               CPU

Apparently the documentation department were having the day off, that time (300b0e8)

afbjorklund commented 2 years ago

Minikube is currently using:

        case "amd64":
                qemuMachine = "" // default
                qemuCPU = ""     // default
        case "arm64":
                qemuMachine = "virt"
                qemuCPU = "cortex-a72"

I believe that ARM is a Raspberry Pi 4 ?

https://en.wikipedia.org/wiki/Raspberry_Pi

gilbahat commented 2 years ago

How about something like this perhaps? that piece of code already has some bugfix magic littering its cases and this did end up being a user-visible bug with a not-so-rare software package...

    case "amd64":
            switch runtime.GOOS {
            case "dragonfly", "freebsd", "netbsd", "openbsd", "windows": // reference to host missing / lima discussion
                    qemuMachine = "" // default
                    qemuCPU = ""     // default
            default:
                    qemuMachine = ""
                    qemuCPU = "host" // recommended by qemu docs and needed for AVX to work
            }
    case "arm64":
            qemuMachine = "virt"
            qemuCPU = "cortex-a72"
afbjorklund commented 2 years ago

Sure, but no need to list all the exotic OS... Just enable it for "linux" and "darwin", when the arch matches ?

gilbahat commented 2 years ago

Sure, but no need to list all the exotic OS... Just enable it for "linux" and "darwin", when the arch matches ?

I agree - if we don't have all the details as it's preferable to totally breaking support for an OS, however exotic. Then again, I'd like to consider 'host' the real default and avoid using it only when there's a reason not to because it has downsides.

afbjorklund commented 2 years ago

It is only semi-supported on Linux and macOS, making qemu work on Windows is a work-in-progress still... NetBSD works (but nobody uses it), and FreeBSD is currently lacking acceleration (ever since kqemu died)

We don't want to suggest that running minikube VM without any hardware acceleration is a good idea, either. It "works" for starting a container or two, but it is not a way to run a cluster - even a single-node one, at that.

afbjorklund commented 2 years ago 
            switch runtime.GOOS {
            case "linux", "darwin":
                    qemuMachine = ""
                    qemuCPU = "host"
            default:
                    qemuMachine = "" // default
                    qemuCPU = ""     // default
            }

Then again, I'd like to consider 'host' the real default and avoid using it only when there's a reason not to because it has downsides.

It is not implemented on very many, there is a matching "max" for the software emulation that has some similar issues.

I thought it was best to leave it as configuration, but running with "host" on macOS and Linux should not be an issue...

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/minikube/issues/14607#issuecomment-1447196563): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.