search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.27k stars 4.87k forks source link

minikube on Apple Silicon / M1 CPU (darwin-arm64) #9224

Closed medyagh closed 3 years ago

medyagh commented 4 years ago

Place holder to document all the issues

❌ Exiting due to DRV_NOT_DETECTED: No possible driver was detected. Try specifying --driver, or see https://minikube.sigs.k8s.io/docs/start/

## Docker can not be installed

Docker Desktop can not be installed on the CPU Incompatible CPU detected



## hyper kit can not be installed

<details>
<pre>
med@meds-mac g1 % brew install hyperkit
Error: dlopen(/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle, 0x0009): symbol '_ffi_prep_closure' not found, expected in flat namespace by '/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle' - /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fiddle.rb:2:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/linkage_checker.rb:6:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula.rb:20:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula_versions.rb:3:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/cmd/update-report.rb:3:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/utils.rb:85:in `require?'
/usr/local/Homebrew/Library/Homebrew/commands.rb:33:in `valid_internal_cmd?'
/usr/local/Homebrew/Library/Homebrew/brew.rb:85:in `<main>'
Error: dlopen(/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle, 0x0009): symbol '_ffi_prep_closure' not found, expected in flat namespace by '/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle' - /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/universal-darwin20/fiddle.bundle
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/fiddle.rb:2:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/linkage_checker.rb:6:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula.rb:20:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/formula_installer.rb:4:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/cmd/install.rb:4:in `<top (required)>'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/rubygems/core_ext/kernel_require.rb:54:in `require'
/usr/local/Homebrew/Library/Homebrew/utils.rb:85:in `require?'
/usr/local/Homebrew/Library/Homebrew/commands.rb:33:in `valid_internal_cmd?'
/usr/local/Homebrew/Library/Homebrew/brew.rb:85:in `<main>'

</pre>
</details>
afbjorklund commented 4 years ago

It seems like VirtualBox is unlikely to be available: https://forums.virtualbox.org/viewtopic.php?f=8&t=98742

afbjorklund commented 4 years ago

Parallells Desktop ($80) has a prototype available: https://www.parallels.com/blogs/apple-silicon-wwdc/

afbjorklund commented 4 years ago

For the linux-arm64 platform (like Raspberry Pi 4), the available minikube options are libvirt (kvm2) and native (none).

We still have to add proper arm64 support to CI, though... Currently only amd64 is being tested. #9205 #6280

afbjorklund commented 4 years ago

Eventually we should also build a KIC base docker image for arm64. KIND is also working on this: https://github.com/kubernetes-sigs/kind/issues/166

https://www.docker.com/blog/getting-started-with-docker-for-arm-on-linux/

https://hub.docker.com/r/arm64v8/ubuntu/

Thanks to QEMU, you can do this on your regular Linux laptop already:

$ arch
x86_64
$ docker run -it arm64v8/ubuntu:focal
root@9937e658add6:/# arch
aarch64

(It uses binfmt_misc and /usr/bin/qemu-aarch64-static under the hood)

That way, once Docker Desktop is done with their VM (LinuxKit) it could run the kicbase arm64 image: https://github.com/docker/for-mac/issues/4733

goafabric commented 3 years ago

Just to let you know, Minikube already works on Apple Silicon M1 !. Together with VFTool https://github.com/evansm7/vftool And a Ubuntu 20 Iso Image, you just can install Docker via apt. Than snatch Minikube ARM64 which is surprisingly there via

curl sudo curl -Lo minikube https://storage.googleapis.com/minikube/releases/latest/minikube-linux-arm64 sudo curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/arm64/kubectl

Drawback you have to work with the None Driver sudo sysctl fs.protected_regular=0 sudo minikube start --driver=none

Be sure to start Vftool with 2 CPUs, not less not more ... And that's it.. just works

afbjorklund commented 3 years ago

Minikube already works on Apple Silicon M1 !.

This issue was actually about darwin (macOS), not so much about the actual hardware (arm64)

It is not so surprising with the arm64 support, and we will have support for more drivers soon...

See #9762 for more ARM links

Together with VFTool https://github.com/evansm7/vftool

Glad to hear that it works, if this has support for libvirt or something we could make it an alternative ?

Currently we are using Docker's HyperKit, and they have been struggling a bit with the ARM support

And no VirtualBox (it's x86 only)

Using QEMU works (on all platforms), but it has some issues with setting up the required networking.

The default user network "works", but minikube is built on a more regular networking model for the VM.

We could also use the "generic" driver, and then you can use any VM - just provide it with the SSH:

See #4733

medyagh commented 3 years ago

@goafabric Glad to see none driver working with vftool (nice discovery)

update: for docker driver I installed the Preview Docker for M1 and it doesnt work for me https://github.com/docker/for-mac/issues/5185

tstromberg commented 3 years ago

With #10115 - the Docker driver will at least work on macOS/arm64 - it will also pave the way for other hypervisors to work, though we'll also need to port the ISO to arm64 (#9228).

We'll also need to add this arch to our release scripts: https://github.com/kubernetes/minikube/issues/10116

sayboras commented 3 years ago

Thanks all for your awesome work, I just try to run minikube in apple silicon, it's working great with docker driver 💯

``` $ go version go version go1.16beta1 darwin/arm64 $ minikube version minikube version: v1.17.0 commit: 94a676a6717732052a7ed96ef00c6f1943af1497 $ kall NAMESPACE NAME READY STATUS RESTARTS AGE kube-system pod/coredns-74ff55c5b-xlr2n 1/1 Running 0 68s kube-system pod/etcd-minikube 1/1 Running 0 81s kube-system pod/kube-apiserver-minikube 1/1 Running 0 81s kube-system pod/kube-controller-manager-minikube 1/1 Running 0 81s kube-system pod/kube-proxy-2gfq5 1/1 Running 0 68s kube-system pod/kube-scheduler-minikube 1/1 Running 0 81s kube-system pod/storage-provisioner 1/1 Running 2 75s NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.96.0.1 443/TCP 85s kube-system service/kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP,9153/TCP 82s NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 82s NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system deployment.apps/coredns 1/1 1 1 83s NAMESPACE NAME DESIRED CURRENT READY AGE kube-system replicaset.apps/coredns-74ff55c5b 1 1 1 68s ```
smithaitufe commented 3 years ago

Thanks all for your awesome work, I just try to run minikube in apple silicon, it's working great with docker driver 💯

$ go version
go version go1.16beta1 darwin/arm64

$ minikube version
minikube version: v1.17.0
commit: 94a676a6717732052a7ed96ef00c6f1943af1497

$ kall
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-74ff55c5b-xlr2n            1/1     Running   0          68s
kube-system   pod/etcd-minikube                      1/1     Running   0          81s
kube-system   pod/kube-apiserver-minikube            1/1     Running   0          81s
kube-system   pod/kube-controller-manager-minikube   1/1     Running   0          81s
kube-system   pod/kube-proxy-2gfq5                   1/1     Running   0          68s
kube-system   pod/kube-scheduler-minikube            1/1     Running   0          81s
kube-system   pod/storage-provisioner                1/1     Running   2          75s

NAMESPACE     NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP                  85s
kube-system   service/kube-dns     ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   82s

NAMESPACE     NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
kube-system   daemonset.apps/kube-proxy   1         1         1       1            1           kubernetes.io/os=linux   82s

NAMESPACE     NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns   1/1     1            1           83s

NAMESPACE     NAME                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/coredns-74ff55c5b   1         1         1       68s

How did you achieve it? Mind documenting here the process you followed?

I have the Apple MacBook Pro 13".

galishmann commented 3 years ago 

➜  ~ minikube start --driver=docker
😄  minikube v1.17.0 on Darwin 11.1
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=3881MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.2 ...
💢  initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:
SIGSEGV: segmentation violation
PC=0x0 m=0 sigcode=0

goroutine 1 [running]:
qemu: uncaught target signal 11 (Segmentation fault) - core dumped

💣  Error starting cluster: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:

😿  minikube is exiting due to an error. If the above message is not useful, open an issue:
👉  https://github.com/kubernetes/minikube/issues/new/choose

❌  Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV
stdout:

stderr:

😿  If the above advice does not help, please let us know: 
👉  https://github.com/kubernetes/minikube/issues/new/choose
sayboras commented 3 years ago

How did you achieve it? Mind documenting here the process you followed?

Oh sorry, I forgot to mention the below points in my system:

``` $ docker info Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., unknown) buildx: Build with BuildKit (Docker Inc., v0.5.1-docker) scan: Docker Scan (Docker Inc., v0.3.5) Server: Containers: 3 Running: 1 Paused: 0 Stopped: 2 Images: 6 Server Version: 20.10.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 4.19.104-linuxkit Operating System: Docker Desktop OSType: linux Architecture: aarch64 CPUs: 4 Total Memory: 1.935GiB Name: docker-desktop ID: PYLF:XCJN:NO3S:L7UQ:AP3I:Z347:NWIH:UIRD:EMW2:YNE7:CMNR:AYQC Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine $ go version go version go1.16beta1 darwin/arm64 $ make go-bindata -nomemcopy -o pkg/minikube/assets/assets.go -pkg assets deploy/addons/... gofmt -s -w pkg/minikube/assets/assets.go go-bindata -nomemcopy -o pkg/minikube/translate/translations.go -pkg translate translations/... gofmt -s -w pkg/minikube/translate/translations.go go build -tags "go_getter_nos3 go_getter_nogcs" -ldflags="-X k8s.io/minikube/pkg/version.version=v1.17.0 -X k8s.io/minikube/pkg/version.isoVersion=v1.17.0 -X k8s.io/minikube/pkg/version.isoPath=minikube/iso -X k8s.io/minikube/pkg/version.gitCommitID="94a676a6717732052a7ed96ef00c6f1943af1497" -X k8s.io/minikube/pkg/version.storageProvisionerVersion=v4" -o out/minikube k8s.io/minikube/cmd/minikube $ ./out/minikube version minikube version: v1.17.0 commit: 94a676a6717732052a7ed96ef00c6f1943af1497 $ ./out/minikube start --driver=docker 😄 minikube v1.17.0 on Darwin 11.1 (arm64) ✨ Using the docker driver based on user configuration 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 🔥 Creating docker container (CPUs=2, Memory=1981MB) ... 🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.2 ... ▪ Generating certificates and keys ... ▪ Booting up control plane ... ▪ Configuring RBAC rules ... 🔎 Verifying Kubernetes components... 🌟 Enabled addons: storage-provisioner, default-storageclass 🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default ```
kientran commented 3 years ago

Side note for those wanting to try this. The M1 build Go-lang formula is available in the M1 Homebrew preview with brew install go. While brew installs 1.15.7, it's actually 1.16beta1 so you don't need to clone and build it separately.

afbjorklund commented 3 years ago

Trying to find the upstream (moby) issue for the qemu segfault, since it seems to be a known issue with this Docker preview...

qemu: uncaught target signal 11 (Segmentation fault) - core dumped

But can mostly find user reports, that are all reporting crashes. Doesn't seem to be anything in https://github.com/moby/qemu

https://github.com/docker/for-mac/labels/area%2Fm1

choudharyanurag commented 3 years ago

@sayboras I did a make on the minikube code but it is not working for me.

MacBook-Pro:minikube anurag$ ./out/minikube start 😄 minikube v1.17.0 on Darwin 11.1 ✨ Using the docker driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🚜 Pulling base image ... 🏃 Updating the running docker "minikube" container ... 🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.2 ... 💢 initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV stdout:

stderr: qemu: uncaught target signal 11 (Segmentation fault) - core dumped

💣 Error starting cluster: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV stdout:

stderr: SIGSEGV:

😿 minikube is exiting due to an error. If the above message is not useful, open an issue: 👉 https://github.com/kubernetes/minikube/issues/new/choose

❌ Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.20.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 139 from signal SEGV stdout:

stderr: SIGSEGV:

😿 If the above advice does not help, please let us know: 👉 https://github.com/kubernetes/minikube/issues/new/choose

sasha7 commented 3 years ago

I've installed arm version of minikube via brew brew install minikube --build-from-source and latest docker preview beta 7, 

❯ minikube start --driver=docker
😄  minikube v1.17.0 on Darwin 11.1 (arm64)
    ▪ MINIKUBE_ACTIVE_DOCKERD=minikube
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.20.2 preload ...
    > preloaded-images-k8s-v8-v1....: 514.92 MiB / 514.92 MiB  100.00% 9.47 MiB
🔥  Creating docker container (CPUs=2, Memory=1981MB) ...
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.2 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

dashboard was loading as well:

❯ minikube dashboard
🔌  Enabling dashboard ...
🤔  Verifying dashboard health ...
🚀  Launching proxy ...
🤔  Verifying proxy health ...
🎉  Opening http://127.0.0.1:58958/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...

Besides doing that, I was able to deploy a simple golang hello world container to it and expose as a service, worked like a charm!

❯ k get all
NAME                           READY   STATUS    RESTARTS   AGE
pod/ms-test-7849ffc897-f8hcr   1/1     Running   0          4m25s
pod/ms-test-7849ffc897-ltjcr   1/1     Running   0          4m22s

NAME                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
service/kubernetes   ClusterIP   10.96.0.1      <none>        443/TCP          16m
service/ms-test      NodePort    10.111.66.75   <none>        8080:31665/TCP   6m40s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ms-test   2/2     2            2           6m40s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/ms-test-5979675bfb   0         0         0       6m40s
replicaset.apps/ms-test-7849ffc897   2         2         2       4m25s

NAME                                          REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/ms-test   Deployment/ms-test   <unknown>/50%   2         5         2          6m40s
brhamill commented 3 years ago

@sasha7 - I tried running the "brew install minikube --build-from-source" and it couldn't get through the process.

Kept giving me the following error. Any thoughts?

image

tstromberg commented 3 years ago

We'll need to update our docs today, but as of last week, folks can install the latest release of minikube for Apple Silicon using:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64
sudo install minikube-darwin-amd64 /usr/local/bin/minikube
brhamill commented 3 years ago

@tstromberg - I attempted to run the commands on my M1. When starting minikube I had the following error. Any ideas? Thanks!

image

tstromberg commented 3 years ago

@brhamill - I've not yet seen a SEGV. Can you please open a new issue? That seems like an unstable VM environment to me, so please check that you are on the latest Docker for M1 build.

choudharyanurag commented 3 years ago

same as @brhamill

image

@tstromberg : Probably I am on the latest build of the Docker for M1 preview : Following is my docker info :

Client: Context: default Debug Mode: false Plugins: app: Docker App (Docker Inc., unknown) buildx: Build with BuildKit (Docker Inc., v0.5.1-docker) scan: Docker Scan (Docker Inc., v0.3.5)

Server: Containers: 1 Running: 1 Paused: 0 Stopped: 0 Images: 2 Server Version: 20.10.1 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Cgroup Version: 1 Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc Default Runtime: runc Init Binary: docker-init containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff init version: de40ad0 Security Options: seccomp Profile: default Kernel Version: 4.19.104-linuxkit Operating System: Docker Desktop OSType: linux Architecture: aarch64 CPUs: 4 Total Memory: 1.935GiB Name: docker-desktop ID: MGZK:F6IX:3OU4:E2WG:AI74:AA5S:TVXR:M2MN:LWPL:5SER:N3ZA:KYQB Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: true Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine

brhamill commented 3 years ago

@tstromberg - the commands you provided:

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-darwin-arm64 sudo install minikube-darwin-amd64 /usr/local/bin/minikube

Should the second line be: sudo install minikube-darwin-arm64 /usr/local/bin/minikube instead? Since it's arm instead of amd?

I made that change and then attempted a "minikube start" I got a zsh permissions error. Am I doing something incorrect with my approach?

zsh: permission denied: minikube

brhamill commented 3 years ago

I was finally able to get minikube running well on my M1 Silicon.

This YT link lays it all out in a very straight-forward way. Hope it helps others...

https://www.youtube.com/watch?v=IjgrwYzVlMw

cheslijones commented 3 years ago

I was finally able to get minikube running well on my M1 Silicon.

This YT link lays it all out in a very straight-forward way. Hope it helps others...

https://www.youtube.com/watch?v=IjgrwYzVlMw

Thanks for sharing. I played with v1.17.1 last night and while I could get my cluster to spin up with skaffold I couldn't figure out how to access it as minikube addons enable ingress didn't work.

--vm-driver=docker seems to be the only supported driver. I've seen people recommend this driver in the past, but neither they nor the documentation actually explains how to use it so the end result has always been the same: my cluster spins up, but I can't access the running application from a browser--doesn't do any good (and port-forward is not a realistic alternative, which someone inevitably recommends doing).

Video seems to actually explain it. I'll give this a shot tonight.

choudharyanurag commented 3 years ago

I was finally able to get minikube running well on my M1 Silicon. This YT link lays it all out in a very straight-forward way. Hope it helps others... https://www.youtube.com/watch?v=IjgrwYzVlMw

Thanks for sharing. I played with v1.17.1 last night and while I could get my cluster to spin up with skaffold I couldn't figure out how to access it as minikube addons enable ingress didn't work.

--vm-driver=docker seems to be the only supported driver. I've seen people recommend this driver in the past, but neither they nor the documentation actually explains how to use it so the end result has always been the same: my cluster spins up, but I can't access the running application from a browser--doesn't do any good (and port-forward is not a realistic alternative, which someone inevitably recommends doing).

Video seems to actually explain it. I'll give this a shot tonight.

minikube tunnel

afbjorklund commented 3 years ago

The Docker Desktop networking model doesn't offer access to the VM, so that basically leaves port forwarding (unfortunately)

https://docs.docker.com/docker-for-mac/networking/

cheslijones commented 3 years ago

Well, did get it working a couple different ways:

  1. The way outlined in the video which is mostly from here. Then had to do minikube service example-service --url to get the actual URL: 
    🏃  Starting tunnel for service example-service.
|-----------|-----------------|-------------|------------------------|
| NAMESPACE |      NAME       | TARGET PORT |          URL           |
|-----------|-----------------|-------------|------------------------|
| default   | example-service |             | http://127.0.0.1:51619 |
|-----------|-----------------|-------------|------------------------|
  2. Also creating a LoadBalander based on the documentation here. After that, minikube tunnel did come to life as opposed to sitting there indefinitely with no console output. But, wasn't able to access the running application from browser. Had to do the same thing as the previous: 
    minikube service client-deployment-dev --url
🏃  Starting tunnel for service client-deployment-dev.
|-----------|-----------------------|-------------|------------------------|
| NAMESPACE |         NAME          | TARGET PORT |          URL           |
|-----------|-----------------------|-------------|------------------------|
| default   | client-deployment-dev |             | http://127.0.0.1:52989 |
|-----------|-----------------------|-------------|------------------------|

Checking the LoadBalancer, it is indefinitely pending on getting an EXTERNAL-IP:

NAME                            TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
client-cluster-ip-service-dev   ClusterIP      10.96.33.149   <none>        3000/TCP         35m
client-deployment-dev           LoadBalancer   10.104.53.56   <pending>     8080:30700/TCP   11m
example-service                 NodePort       10.101.7.132   <none>        3000:32764/TCP   27m
kubernetes                      ClusterIP      10.96.0.1      <none>        443/TCP          40m

Doesn't appear to be any issues in the describe:

Name:                     client-deployment-dev
Namespace:                default
Labels:                   app.kubernetes.io/managed-by=skaffold
                          skaffold.dev/run-id=d46b55ce-ca0f-4615-b9e2-22166fdf0b93
Annotations:              <none>
Selector:                 component=client,environment=development
Type:                     LoadBalancer
IP:                       10.104.53.56
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30700/TCP
Endpoints:                172.17.0.3:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

At any rate, with any of these, I'd have to make a service for each deployment--I have one / (client), /api, /admin. I can see that presenting issues similar to what I've faced using port-forward a call from / to the /api will append localhost:3000, for example, but /api is at localhost:5000, so then it says it can't find the API at localhost:3000/api.

Definitely a lot more straight forward with KVM2, Hyperkit, etc. drivers and using an ingress controller like ingress-nginx. Is there any time frame for Virtualization.Framework support, lol?

cheslijones commented 3 years ago

I did manage to get minikube tunnel and ingress-nginx to work together.

I just needed to enable ingress-nginx for Docker Desktop with:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/cloud/deploy.yaml

I started my cluster with skaffold.

When that was running I ran minikube tunnel and it found the ingress-nginx-controller saying:

❗  The service ingress-nginx-controller requires privileged ports to be exposed: [80 443]
🔑  sudo permission will be asked for it.
🏃  Starting tunnel for service ingress-nginx-controller.

Didn't have to expose any, or each, of my microservices and routing seems to be working fine between them where they refer to one another.

So using --driver=docker is basically one more step than hyperkit, kvm, etc.: you need to run minikube tunnel. I've had issues with doing that in the past, I think in WSL2, but seems to work with macOS, minikube 1.17.1, Docker M1 preview build, etc.

franciscosuca commented 3 years ago

I was finally able to get minikube running well on my M1 Silicon.

This YT link lays it all out in a very straight-forward way. Hope it helps others...

https://www.youtube.com/watch?v=IjgrwYzVlMw

This answer and the new Docker Desktop preview 3.1.0 (with K8s included) helped me to install it successfully.

Thanks!

tstromberg commented 3 years ago

Support for M1 launched, so I'm closing this in preference to individual issues being opened.

NitichaiSawangsai commented 3 years ago

You try this command

 arch -x86_64 brew install hyperkit & minikube start --vm=true
ayepRahman commented 3 years ago

You try this command

arch -x86_64 brew install hyperkit & minikube start --vm=true

Have not try this, but are you saying iam able to run minikube on arm64 with --docker=hyperkit without the needing of port forwarding?

slmingol commented 2 years ago

Confirming that Docker's official .dmg worked fine for me along w/ brew install minikube

kamilgregorczyk commented 2 years ago

Is there a way of using minikube without the docker for mac app on M1? it's license became kind of repelling, hyperkit doesn't seem to work on M1.

chicks-net commented 2 years ago

Is there a way of using minikube without the docker for mac app on M1? it's license became kind of repelling, hyperkit doesn't seem to work on M1.

I've gotten minikube going with podman 4.0.2 on an M1 Mac. More details are in this stackoverflow answer.

I just filed #13855 to see if we can clean this up a bit more.

temikelani commented 2 years ago

use docker - install docker desktop for Mac M1 and run

source: https://medium.com/@seohee.sophie.kwon/how-to-run-a-minikube-on-apple-silicon-m1-8373c248d669

jnardiello commented 1 year ago

Virtualbox on apple silicon (M1/M2) is coming along and is now available on the developer preview version. Any chance minikube is going to support that? Have you ever discussed it or done any work on it?

Rajat-Dabade commented 1 year ago

Virtualbox on apple silicon (M1/M2) is coming along and is now available on the developer preview version. Any chance minikube is going to support that? Have you ever discussed it or done any work on it?

I have tried running it with a developer preview version of Virtualbox on M1 but hard luck.

minikube start --driver=virtualbox                                                                      ok  4s  18.14.2 node 
😄  minikube v1.29.0 on Darwin 12.6 (arm64)
✨  Using the virtualbox driver based on user configuration

❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'virtualbox' is not supported on darwin/arm64
lowang-bh commented 1 year ago

Virtualbox on apple silicon (M1/M2) is coming along and is now available on the developer preview version. Any chance minikube is going to support that? Have you ever discussed it or done any work on it?

I have tried running it with a developer preview version of Virtualbox on M1 but hard luck.

minikube start --driver=virtualbox                                                                      ok  4s  18.14.2 node 
😄  minikube v1.29.0 on Darwin 12.6 (arm64)
✨  Using the virtualbox driver based on user configuration

❌  Exiting due to DRV_UNSUPPORTED_OS: The driver 'virtualbox' is not supported on darwin/arm64

how about minikube start --driver=docker?

darkn3rd commented 1 year ago

And no VirtualBox (it's x86 only)

Actually, there's an arm64 virtualbox now: https://www.virtualbox.org/wiki/Download_Old_Builds_7_0

afbjorklund commented 1 year ago

The arm64 VirtualBox runs Intel emulation, not what you want for minikube

https://forums.virtualbox.org/viewtopic.php?t=107344

desylva commented 2 days ago

And no VirtualBox (it's x86 only)

Actually, there's an arm64 virtualbox now: https://www.virtualbox.org/wiki/Download_Old_Builds_7_0

❌ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: creating host: create: precreate: Docker Machine can be used only with Parallels Desktop Pro or Business edition. You use: standard edition

Argh...