Provider: utm qemu libvirt

[georgestephanis]

Was chatting with @evertiro in slack about this, logging some potentially useful links -- to the best of my knowledge, all linked projects here are free and open source.

UTM can run virtual machines beautifully on the M1 Macs using hypervisor -- https://mac.getutm.app/ -- it also has a version that runs on iPhones and iPads, but that's less salient to our purposes. https://github.com/utmapp/UTM/

There's been some questions on their repository about adding support for Vagrant:

https://github.com/utmapp/UTM/issues/2496

with some planned changes that may make it more doable:

https://github.com/utmapp/UTM/issues/2280

but it is noted there that UTM is largely a wrapper about QEMU.

UTM recently added some functionality to allow exporting the QEMU arguments out of UTM, if it simplifies anyone's workflow when doing initial explorations: https://github.com/utmapp/UTM/commit/48a8fc74fff07e12199670da2107fd4269a6d70f

QEMU -- https://www.qemu.org/ -- https://gitlab.com/qemu-project/qemu/ -- is a generic and open source machine emulator and virtualizer.

libvirt -- https://libvirt.org/ -- also works as a wrapper around qemu, and reportedly runs just fine on apple silicon! https://doesitarm.com/formula/libvirt/

It also has a vagrant provider: https://github.com/vagrant-libvirt/vagrant-libvirt

Anyway, just wanted to dump some starting points and research here for some assorted ways we could get VVV running smoothly on Apple Silicon with open source tools.

[welcome[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template and include your OS/Vagrant/VVV versions! Don't forget you can get support in the VVV slack at https://varyingvagrantvagrants.org/docs/en-US/slack/

VVV

VVV Slack Workspace

Join the VVV Slack Workspace

[georgestephanis]

Possibly salient notes from someone else:

https://gist.github.com/aserhat/91c1d5633d395d45dc8e5ab12c6b4767

Gist

QEMU and HVF

QEMU and HVF. GitHub Gist: instantly share code, notes, and snippets.

[evertiro]

I don't think this is necessarily an M1 or even Mac-specific thing. QEMU/libvirt are available for all three supported operating systems, so if we can make it work this could be our first completely free cross-platform provider. I say it's worth a shot!

[evertiro]

I'd like to point out that libvirt already has a stable vagrant provider. I'd say that's probably the easiest place to start.

[tomjn]

For MacOS we ideally want something using Apples hypervisor, which is what VMWare Docker and Parallels use

[georgestephanis]

Based on what research I'd done, it seems qemu does also use hypervisor.

UTM explicitly says it does as well.

UTM employs Apple's Hypervisor virtualization framework to run ARM64 operating systems on Apple Silicon at near native speeds.

and

Under the hood of UTM is QEMU, a decades old, free and open source emulation software that is widely used and actively maintained.

According to https://www.naut.ca/blog/2020/08/26/ubuntu-vm-on-macos-with-libvirt-qemu/:

QEMU on the other hand has had support for Hypervisor.Framework since 2018.

[tomjn]

then we need the following:

• an arm64 Ubuntu 20 box
• a provider plugin

This also assumes no efforts towards containerised VVV or native VVV

[tomjn]

This libvirt box looks the most promising at the moment on vagrant clouds box catalog:

https://app.vagrantup.com/travismccollum/boxes/u20_arm64

There's also xhyve, which has been around a while and uses a port of bhyve to interface with Apples hypervisor framework

https://github.com/machyve/xhyve

We would need to build our own box though, and I couldn't find any information in a brief look for Apple Silicon compatibility

Vagrant Cloud by HashiCorp

Vagrant Cloud by HashiCorp

Vagrant Cloud by HashiCorp

GitHub

machyve/xhyve

xhyve, a lightweight OS X virtualization solution. Contribute to machyve/xhyve development by creating an account on GitHub.

[tomjn]

If someone was willing to roll up their sleeves with xcode and some ruby, they could bypass all of these and build something specific to VVV+MacOS

This issue has a lot of interesting things in it https://github.com/machyve/xhyve/issues/206

[evertiro]

I mean eventually. Though I'm not normally a fan of writing ruby.

[tomjn]

Multipass by canonical may be a viable alternative, it uses Apples hypervisor behind the scenes on MacOS, kvm on linux, and hyper-v on Windows. There's a Vagrant plugin but it hasn't been touched in 14 months.

We would need to figure out our own box system though, or if boxes even make sense for multipass, there's nothing in a search for multipass on vagrant cloud

[GregOriol]

I'm very interested in this: I'd like to run an ubuntu x86 via vagrant on an M1 mac. The guest system runs with UTM/qemu, what is missing is the interaction with vagrant. It would be very useful in that setup because some packages I use inside the vagrant box (which is similar to the production environment) don't have available binaries for ubuntu arm...

[tomjn]

It may be possible to build a primitive rigged vagrant plugin by building a VM in UTM that can be downloaded that's preconfigured then using utm:// links but it'll be hacky and probably have a GUI window:

https://github.com/utmapp/UTM/wiki/URL-Scheme-%E2%80%93-Automation-for-UTM-app-&-VMs

Also relevant: https://github.com/utmapp/UTM/discussions/3618

Possibly require us to mess with their plist format, it would require a bit of effort but it looks doable. The concern I have is shared/mounted folders may be a pain in the arse to figure out.

What I would like is someone who can do the following:

• setup a UTM virtual machine on Apple Silicon running headless Ubuntu 20 with 2 or 3 shared folders
• export that VM somewhere, including any plist files from UTM for inspection

This way we can try to reverse engineer some things and figure stuff out

cc @GregOriol @Mamaduka

[trinitronx]

libvirt -- https://libvirt.org/ -- also works as a wrapper around qemu, and reportedly runs just fine on apple silicon! https://doesitarm.com/formula/libvirt/

It also has a vagrant provider: https://github.com/vagrant-libvirt/vagrant-libvirt

This is quite possible to do right now with fully native arm64 libvirt & qemu binaries + gem native arm64 extensions. The main obstacle is getting vagrant itself to run as a gem on the Apple Silicon M1, because there is no official arm64 release yet.

If using macOS VM guests (yes, it's possible, but there's some tricks to it... e.g. they need NVRAM, and some magic incantations. It also requires a few more unreleased gem patches.)

Status update on all the dependencies needed to get this working with the libvirt provider can be found here

EDIT: The other nice feature with using qemu + libvirt is that you can launch VMs on a remote host machine over the network with qemu+ssh:// protocol.

[tomjn]

@trinitronx vagrant runs on Apple silicon, I and others have been using M1 type machines with Parallels for months.

If using macOS VM guests (yes, it's possible, but there's some tricks to it... e.g. they need NVRAM, and some magic incantations. It also requires https://github.com/vagrant-libvirt/vagrant-libvirt/issues/1371#issuecomment-1120223051.)

VVV uses debian/Ubuntu type VMs, we aren't interested in MacOS guests.

because https://github.com/hashicorp/vagrant/issues/12548.

I don't see much progress here, especially with Vagrant being ported to Golang. Arm64 builds on golang should be significantly easier to get going, and I expect any Arm support vagrant provides to happen there first.

It makes dealing with this that much more difficult as writing a UTM provider for example means writing it in ruby with all the deprecations coming in 2.3 and 2.4, or, writing it in golang and having to wait for 2.3 for limited documentation to be released.

---

Another primary problem is that if we send users to install VirtualBox they get an installer, run it, reboot, done. But if we send them to install QEMU it's much more difficult. That's why we hoped UTM would resolve the issue. A lot of VVV users are not technical, don't know how to program, and struggle with the most basic aspects of CLI usage.

E.g. if we send users to libvirt.org or just say "you need libvirt" then ¯_(ツ)_/¯ we won't get any users, it's gibberish, years beyond the technical skill of 99% of the users who would need such a provider, making the whole endeavour pointless.

[georgestephanis]

Just noting that I'd spotted this today as a possible way of letting vagrant use qemu directly as a vagrant plugin

https://github.com/ppggff/vagrant-qemu

GitHub

GitHub - ppggff/vagrant-qemu: Use Vagrant to manage machines using QEMU. Test with Apple Silicon / M1 and CentOS aarch64 image

Use Vagrant to manage machines using QEMU. Test with Apple Silicon / M1 and CentOS aarch64 image - GitHub - ppggff/vagrant-qemu: Use Vagrant to manage machines using QEMU. Test with Apple Silicon /...