starting minikube on arm64 Mac with Parallels

[umeshch1]

Steps to reproduce the issue:

1.❯ minikube start 😄 minikube v1.19.0 on Darwin 11.2.3 (arm64) ✨ Automatically selected the parallels driver

❌ Exiting due to DRV_UNSUPPORTED_OS: The driver 'parallels' is not supported on darwin/arm64

Full output of failed command:

❯ minikube start --driver=parallels

😄 minikube v1.19.0 on Darwin 11.2.3 (arm64) ✨ Using the parallels driver based on user configuration

❌ Exiting due to DRV_UNSUPPORTED_OS: The driver 'parallels' is not supported on darwin/arm64

Full output of minikube start command used, if not already included:

Optional: Full output of minikube logs command:

After installing minikube on M1 MacBook Air through homebrew, started using above command. Getting error for the drivers parallels. I have parallels installed as evaluation. Installation logs: ❯ brew install minikube ==> Downloading https://ghcr.io/v2/homebrew/core/minikube/manifests/1.19.0 ######################################################################## 100.0% ==> Downloading https://ghcr.io/v2/homebrew/core/minikube/blobs/sha256:8704e50e9352938 ==> Downloading from https://pkg-containers-az.githubusercontent.com/ghcr1/blobs/sha25 ######################################################################## 100.0% ==> Pouring minikube--1.19.0.arm64_big_sur.bottle.tar.gz ==> Caveats zsh completions have been installed to: /opt/homebrew/share/zsh/site-functions ==> Summary 🍺 /opt/homebrew/Cellar/minikube/1.19.0: 8 files, 58.8MB

[afbjorklund]

Is docker-machine-driver-parallels available for arm64 ? If so, the "support" list needs updating...

// supportedDrivers is a list of supported drivers on Darwin.
var supportedDrivers []string = func() []string {
        if runtime.GOARCH == "arm64" {
                // on darwin/arm64 only docker and ssh are supported yet
                return []string{
                        Docker,
                        SSH,
                }
        }
        return []string{
                VirtualBox,
                Parallels,
                VMwareFusion,
                HyperKit,
                VMware,
                Docker,
                SSH,
        }
}()

See https://github.com/Parallels/docker-machine-parallels/issues/99

[afbjorklund]

This also needs an ISO to be available for arm64, but that is something that is planned.

You should be able to start an Ubuntu VM with Parallels, and then use the SSH driver ?

---

If Parallels supports Vagrant (as a provider) you could use that with the Vagrantfile (https://github.com/kubernetes/minikube/issues/10089#issuecomment-755183661)

Vagrant.configure("2") do |config|
  config.vm.box = "ubuntu/focal64"

  config.vm.network "private_network", ip: "192.168.50.4"

  config.vm.provider "virtualbox" do |vb|
    vb.cpus = 2
    vb.memory = "2048"

    # focal64 bug: https://bugs.launchpad.net/cloud-images/+bug/1829625
    vb.customize [ "modifyvm", :id, "--uartmode1", "file", File::NULL ]
  end

  config.vm.provision "shell", inline: <<-SHELL
    apt-get update
    apt-get install -y conntrack 
  SHELL
end

EDIT: Apparently, you need to use "bento/ubuntu-20.04" for provider "parallels"

• https://app.vagrantup.com/ubuntu/boxes/focal64
• http://parallels.github.io/vagrant-parallels/docs/boxes/
• https://app.vagrantup.com/bento/boxes/ubuntu-20.04

Otherwise the regular Ubuntu LTS Server installation ISO and clicking through the wizard should work too.

https://cdimage.ubuntu.com/releases/20.04/release/ubuntu-20.04.2-live-server-arm64.iso

Currently you still have to install the container runtime and some other requirements (conntrack) yourself.

In the future, minikube is supposed to be able to provision those itself (but it is currenly broken) #10883

[ilya-zuyev]

atm, only docker driver is supported on M1

[afbjorklund]

blocked on #9228 for the minikube.iso

[afbjorklund]

We could need some help from Parallels users and developers, to support this VM driver.

There are no integration tests for VMware or Parallels, so they rely on community support...

[mprimeaux]

@afbjorklund I am on an M1 + Parallels so am happy to help. Would you mind catching me up a bit on what gaps remain?

[afbjorklund]

@mprimeaux if you can verify if this ISO works (after expanding), then we know we can use it as the base for the new:

https://github.com/afbjorklund/minimal-buildroot/releases/download/0.2/boot-arm64.iso.gz (it requires UEFI to boot)

[mprimeaux]

@afbjorklund Thanks for the gz. I created an ISO with the files and booted in Parallels. Here's what I see.

[afbjorklund]

Not so much sure about "create", it was more like "insert" (CD) ? Just meant that you needed to gunzip it first.

Like so: https://kb.parallels.com/en/122842 (using the boot-arm64.iso)

Anyway, it is supposed to show you the GRUB menu first, then boot the kernel, and then dump you at a prompt:

You can login as root at the serial console. This "minimaĺ" ISO doesn't have much software, though.

[mprimeaux]

@afbjorklund Thanks for the detail. Apologies. Perhaps I misunderstood.

The link to the arm64 ISO you provided decompresses to a folder structure that doesn't contain an ISO and so I thought you preferred me to create and ISO using the linked gz as the source for boot-arm64.iso.

[afbjorklund]

Ah, maybe it got clever and mounted the ISO rather than just decompressing the file (from .gz).

The file structure on the CD looks something like this:

/mnt/iso
├── boot
│   ├── initrd.img
│   └── vmlinuz
├── boot.catalog
└── EFI
    └── BOOT
        ├── bootaa64.efi
        ├── efiboot.img
        └── grub.cfg

3 directories, 6 files

But the file is supposed to be used as-is by the VM, it is a bootable Live CD image (just compressed)

---

Here is what it looks like in a more GUI tool like VirtualBox (here booting the boot-amd64.iso version)

https://user-images.githubusercontent.com/10364051/129453441-5698c33f-0336-428e-a70f-2bbfc4557d68.mp4

[mprimeaux]

I created a 'blank' virtual machine and then attempted to attach the compressed ISO as per the Parallels KB article you linked above. Here's the 'play by play' of the failed attempt.

The ISO structure that I have, of course, matches the one you show in your previous reply. I'll dig around more.

[afbjorklund]

You will need to decompress the file.

gunzip boot-arm64.iso.gz

I think next time I will not compress it 😀

(the kernel and initrd are already compressed)

[mprimeaux]

@afbjorklund Progress :)

https://user-images.githubusercontent.com/293512/129458785-d8a14369-7753-48ae-98b9-06be6fd16a4a.mov

Not very exciting but at least grub is coming up. Just hangs after that. I'll let it run a bit more to see if anything 'times out' but not holding my breath.

Also, I can confirm that 'CD/DVD' is first in the boot order.

[afbjorklund]

There is a supposed to be a 5 second countdown, but if you touch the keys it will wait for a "return" before continuing

But it seems like something is not working with the boot sequence, that is a bit disappointing but we can wait for the iso.

  Booting `Buildroot'

EFI stub: Booting Linux Kernel...
EFI stub: Using DTB from configuration table
EFI stub: Exiting boot services and installing virtual address map...
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd034]
...
[    2.312224] Run /init as init process

So for some reason, EFI is failing to start Linux - while running in Parallels.

[mprimeaux]

A bit disappointing for sure.

Regarding,

...that is a bit disappointing but we can wait for the iso.

Which ISO?

[afbjorklund]

Which ISO?

See #9228

The real minikube.iso (for arm64), currently struggling a bit with getting the amd64 variant built in CI.

I think most of the packages have been prepared, so it's more a matter of integrating this new ISO...

i.e. split the buildroot config file and the linux config file into two separate ones, for amd64 and arm64

The current ISO uses isolinux which is x86_64-only, most likely we will only change for aarch64

---

if you want to continue on the parallels side, the driver needs building for both architectures now

and maybe you (or someone else) will be able to find out the issue with the "minimal" iso boot ?

We will see what the test results on Apple M1 when running the ISO with Docker HyperKit is.

And there is supposed to be HVF hardware support for arm64 too in a QEMU release "soon"

• https://github.com/containers/podman/blob/main/docs/tutorials/mac_experimental.md

• https://mac.getutm.app/

[afbjorklund]

Sounds great! If possible, it would be nice to know if the amd64 UEFI image works when running on Intel ?

https://github.com/afbjorklund/minimal-buildroot/releases/download/0.2/boot-amd64.iso.gz

As I mentioned, I tested it with VirtualBox (after ticking the "EFI" box*), so I think it works with Parallels too...

* this box, I think it is enabled by default in Parallels (says https://kb.parallels.com/en/122735 at least)

I'm not sure yet if we will change both architectures to use GRUB and UEFI, or if we still stick with isolinux.

It should also be possible to keep a hybrid BIOS/UEFI image, if we want to be compatible (for the x86_64).

[mprimeaux]

@afbjorklund I can confirm the boot-amd64.iso.gz works in Parallels 17 on macOS (Intel). As per the KB, I needed to set vm.bios.efi=1 explicitly in Boot Flags.

https://user-images.githubusercontent.com/293512/129480225-edeef7be-90ca-4091-814a-7f7d28857cfc.mov

Interestingly, when setting the same boot flag in Parallels 17 on Apple Silicon for the ARM image, I receive the following error, which causes a Technical Data Report to be generated.

VM Log vm.log

[mprimeaux]

Is there any testing that I can assist with for this issue?

[sharifelgamal]

We're currently working on an arm64 minikube ISO, it should be available for testing in the next few weeks.

[mprimeaux]

@sharifelgamal Any updates?

[sharifelgamal]

See #9228 for all updates on the development of the minikube VM image for arm64.

[tidbeck]

Could this be fixed now when #9228 is resolved?

[gilbahat]

For comparison purposes - buildroot 2021 uses linux 5.10 while the default supported image that parallels will try to install is using linux 5.13. I strongly suspect that bumping buildroot version to 2022.05 or 2022.02 would resolve this, but I am unable to test due to https://github.com/kubernetes/minikube/issues/14500 and would appreciate help.

[afbjorklund]

buildroot 2021 uses linux 5.10

note that buildroot supports multiple kernels

https://github.com/buildroot/buildroot/blob/2022.02.x/linux/linux.hash

for minikube we have tried to pick LTS kernels

And also to keep it similar to Ubuntu LTS versions... (used for KIC etc)

[intrepidsilence]

As a quick note, Parallels now officially states support for the parallels driver for minikube in Parallels Dektop for Mac Pro Edition as of version 18. However, the brew version of minikube still gives the error about not being support on ARM.

Exiting due to DRV_UNSUPPORTED_OS: The driver 'parallels' is not supported on darwin/arm64

[gilbahat]

As a quick note, Parallels now officially states support for the parallels driver for minikube in Parallels Dektop for Mac Pro Edition as of version 18. However, the brew version of minikube still gives the error about not being support on ARM.

Exiting due to DRV_UNSUPPORTED_OS: The driver 'parallels' is not supported on darwin/arm64

This is intriguing given the experience I had here: https://github.com/kubernetes/minikube/issues/14661 that issue seems to suggest the problems with supporting parallels originated at least partially with how minikube built its iso, and that hasn't changed. I would love to take a look.

[afbjorklund]

That exit should probably be avoided, it's not very helpful anyway (even it might still be true, about "supported")

It is hardcoded here: https://github.com/kubernetes/minikube/blob/master/pkg/minikube/driver/driver_darwin.go

[mprimeaux]

That exit should probably be avoided, it's not very helpful anyway (even it might still be true, about "supported")

Agreed.

I modified the file referenced above and configured minikube to use Parallels as the driver.

I'm using my MBP M1 Max with Parallels Desktop 18 Pro edition.

The KIC image downloaded as expected and the resulting VM started. Unfortunately, an IP addressed couldn't be allocated to the VM so it failed to start after a few 120-second timeouts were exceeded.

I'm traveling so it may be related to my environment. I'll work on testing once back to a stable network environment.

[mprimeaux]

@intrepidsilence The only reference I can find on the Parallels site that mentions minikube is here. Do you have a more precise link to an announcement that you don't mind posting?

I see minikube references github.com/Parallels/docker-machine-parallels but the project hasn't been updated for over two years. This might be fine since it appears we call prlctl.

Of note, I had to deal with a few security prompts (camera and microphone) when the VM first started the boot process. There appears to be an error obtaining an IP lease. Perhaps there's a networking change in Parallels 18 (???)

logs.txt

❯ ./minikube config view                                                                                                                                   ─╯
- cpus: 8
- driver: parallels
- memory: 32768
- container-runtime: containerd

Setting container-runtime to docker also fails with the same behavior.

[intrepidsilence]

@mprimeaux Sure! Check here and look at the comparison table. It is also mentioned in other places on that page.

[mprimeaux]

Thanks for the link. I don't see mention of supporting minikube for any specific CPU architecture but just that it supports minikube.

Integrations With Vagrant/Docker/Packer/Minikube

I wish they had a more specific KB article on precisely the technical configurations they support. I'll see if I can install Parallels 18 on my Intel-based Mac.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[intrepidsilence]

Thanks for the link. I don't see mention of supporting minikube for any specific CPU architecture but just that it supports minikube.

Integrations With Vagrant/Docker/Packer/Minikube

I wish they had a more specific KB article on precisely the technical configurations they support. I'll see if I can install Parallels 18 on my Intel-based Mac.

@mprimeaux : Did you ever find time to test this out?

[mprimeaux]

@intrepidsilence Sorry for not replying. Completely forgot. Yes, I am able to install Parallels 18 on my Intel-based Mac Mini and it works as expected with Minikube (v1.28.0).

[intrepidsilence]

@mprimeaux Thanks! I think we should close this issue.

[platput]

Wait, so does it work now? I mean when I tried it today I still got the same error: Exiting due to DRV_UNSUPPORTED_OS: The driver 'parallels' is not supported on darwin/arm64

[mprimeaux]

@platoputhur In a nutshell, Minikube is able use Parallels as a driver on macOS Intel but not on macOS Apple Silicon at the moment. Currently, Minikube running on ARM64 explicitly performs a check here. This is not to say it does not work.

If you'd like, I can comment out the check to allow Parallels as a driver on ARM64 to see if it does work. If it does then I can file a PR to remove the check. Please let me know.

[platput]

@mprimeaux thanks for replying. Yeah please do and let me know.

[novski]

My Homebrew installation of minikube 1.28.0 showed this error. I had to set the driver manualy described here: https://minikube.sigs.k8s.io/docs/drivers/docker/ minikube start --driver=docker or permanent: minikube config set driver docker this solved it...

[nikitar]

I had a play with removing the restriction in driver_darwin.go locally and I'm hitting similar image-related issues as @mprimeaux above. Using the latest arm64 iso, it seems that the vm never finishes booting properly and hence it's never added to the dhcp config in /Library/Preferences/Parallels/parallels_dhcp_leases. In parallels 18.1, the vm never gets past the three penguins stage:

I also tested it in Virtual Box, which has just added support for m1 macs. It shows Script Error Status: Unsupported (line number 1) when running bootaa64.efi.

I don't know enough about linux/efi/vm's to debug this, unfortunately. But if somebody has additional pointers, I'd be happy to have a farther look.

I did try building a new image locally, but it seems that building an iso for arm64 is intended to be done on amd64 using gcc-multilib. An image I managed to build locally on m1 mac ended up being only 74mb and broken (it cannot even be mounted).

[MoElaSec]

atm, only docker driver is supported on M1

Did things change by now ?

[afbjorklund]

I also tested it in Virtual Box, which has just added support for m1 macs.

As far as I know, VirtualBox only supports running Intel images in emulation ?

That is not going to be useful for minikube, since running a cluster demands more

[afbjorklund]

I did try building a new image locally, but it seems that building an iso for arm64 is intended to be done on amd64 using gcc-multilib.

It would be possible to modify the docker image to avoid that (i.e. use gcc, on arm64) but the default environment builds the ISO on a debian virtual machine - not using the provided builder image

Upstream image: https://github.com/buildroot/buildroot/blob/2022.02.x/support/docker/Dockerfile

[afbjorklund]

atm, only docker driver is supported on M1

Did things change by now ?

The qemu2 driver is the supported one (also on M1)

https://minikube.sigs.k8s.io/docs/drivers/qemu2/

[gilbahat]

There is a DP of virtualbox emulating arm on m1: https://www.virtualbox.org/wiki/Downloads However given my 'adventures' in https://github.com/kubernetes/minikube/issues/14661 I doubt this would play out any differently than vmware fusion and/or parallels arm, until the build options and boot sequence are fixed.

[afbjorklund]

As far as I can tell, it still only emulates Intel: https://forums.virtualbox.org/viewtopic.php?f=8&t=107344

It would be more interesting with a vz driver, than looking at the oracle virtualbox or docker hyperkit legacy.

[afbjorklund]

I doubt this would play out any differently than vmware fusion and/or parallels arm, until the build options and boot sequence are fixed.

I don't know who will be working on this in the future, but it is something for the 2023 roadmap to be sure...

It needs updating to 2022.02 (possibly 2023.02 ?), and there is a lot of cleanup and deduplication needed as well.

[caerulescens]

@afbjorklund I'm investigating parallels support, and I have a MacBook Pro with an Intel processor and a MacBook Pro with an m2 processor. My daily os is Debian 12 on amd64; I could help in this area if no one else is working on this.