Provider for UTM (VM for M1 processors)

[brtsos]

On new Mac's with M1 processors users can't use Virtualbox and VMware, but UTM works: https://mac.getutm.app/

Could you please add provider for UTM? That will save a lot of developer life :) Thanks!

[lukechilds]

UTM doesn't expose an API AFAIK so I don't think it would be possible to build a provider for it.

However UTM is simply a GUI wrapper around QEMU. An official QEMU provider would indeed be awesome to see for Vagrant, it would provide very wide platform support including Linux, Mac (Intel and arm) and Windows.

There is already a libvirt provider for Vagrant (https://github.com/vagrant-libvirt/vagrant-libvirt) which is able to interface with QEMU via the libvirt API but I haven't had much luck getting it working outside Linux. The libvirt API only exposes a subset of what you can do with QEMU directly so a QEMU provider would still be a great addition IMO.

[osy]

I am a developer for UTM and we are interested in bringing Vagrant support. Is there a high level list of things that is needed to bring support? We support both QEMU and Apple Virtualization.framework so there's consideration in what features we can support that will work well with both.

[ktprograms]

Hi, I also help to develop UTM, and have looked at the custom provider docs for vagrant.

If I understand it correctly, the bare minimum needed for a provider is to:

• Define a box format
• Implement vagrant up to start the VM
• Implement vagrant ssh, which would require some way to get the VM's IP address (since UTM can use Shared or Bridged networking)

Is there anything else that would be needed (at least initially)?

Is implementing vagrant status necessary for a valid provider? The VM state is already stored, we would just need to figure out some way to expose it externally.

Edit: Looking at some example Vagrantfiles using the Virtualbox provider, it seems like configuration of the VM is done in the Vagrantfile (for example CPU cores, memory, networking). This isn't really compatible with UTM because it stores a config.plist in the VM bundle.

Is exposing VM configuration to the Vagrantfile necessary?

[stblassitude]

Looking at some example Vagrantfiles using the Virtualbox provider, it seems like configuration of the VM is done in the Vagrantfile (for example CPU cores, memory, networking). This isn't really compatible with UTM because it stores a config.plist in the VM bundle.

Is exposing VM configuration to the Vagrantfile necessary?

Not a Vagrant expert, just a long time user, and a recent Apple Silicon user, so I would welcome UTM support!

Vagrant users expect that the virtualization application becomes invisible; you don't want to need to touch it at all. You script the VM(s) you need using the Vagrantfile, then Vagrant and the VM provider create the desired VM and give you access (via SSH, port forwardings, custom network config, and shared filesystems). When you're done, you have Vagrant destroy the VM(s) again. It is therefor necessary to have Vagrant being able to specify relevant desired VM parameters, mainly the amount of RAM and the number CPU cores, and network parameters. I have had more complicated setups where I also added additional disks through Vagrant. And while it's possible to use Vagrant to create a VM with a GUI, I think the overwhelming use case is to have a headless server.

Think of Vagrant as a mini IaaS cloud on you laptop.

[fredngo]

@osy @ktprograms I know it's frowned upon to write "me too" messages but I do want to thank you for your work with UTM and really hope you can bring a provider to Vagrant!

Edit: Looking at some example Vagrantfiles using the Virtualbox provider, it seems like configuration of the VM is done in the Vagrantfile (for example CPU cores, memory, networking). This isn't really compatible with UTM because it stores a config.plist in the VM bundle. Is exposing VM configuration to the Vagrantfile necessary?

As for this question -- I don't know the specifics (I'm just a user of Vagrant, not a developer of Vagrant) but as I understand it the idea is that the configuration of the machine is written in the Vagrantfile, and the comnand-line UTM program should accept command-line parameters in order to create the machine with the right config (# core, memory, networking). Once it's created then of course you can save those configs in your internal plist file.

[tomjn]

@osy @ktprograms you're best looking at the parallels provider as an example. However some finer details

I can foresee the following:

• a vagrant plugin will need to be built that adds utm as a provider
• vagrant files and plugins are written in Ruby
• this plugin will interface with utm, Virtualbox does this via calls to vboxmanage and other CLI, Parallels has an equivalent, UTM will likely need the same

As for the Vagrant file, as a writer of a vagrant file I specify the machines I want and some options as documented by the vagrant docs. This is a declarative API, so I never interact with the VM itself, rather I place an order with Vagrant who goes away and cooks up an environment for me. Then I can run provisioners scripts/commands in those environments once they've booted up. The vagrant plugin would be responsible for taking that and issuing the necessary instructions to UTM.

The other factor is how your box will be formatted, I couldn't comment on the details of that.

• Here's the parallels provider: https://github.com/Parallels/vagrant-parallels
• Vagrant plugin basics: https://www.vagrantup.com/docs/plugins/development-basics
• Here's vagrant provider docs: https://www.vagrantup.com/docs/plugins/providers
• Packaging and distributing plugins: https://www.vagrantup.com/docs/plugins/packaging

I believe with this you could get a very simple level of support operational quickly depending on UTM's APIs, and I'd suggest encouraging open-source contributions with docs. The finer points of shared/mounted folders and networking options may take longer to build, though provisioners should be quicker I would expect.

The biggest obstacle I can see right now is that there is no documented API for UTM or CLI application advertised. If this existed someone would probably have built a basic vagrant plugin by now, but instead it looks like a pure GUI Mac application

[jamesmortensen]

What's the advantage of having UTM be the provider instead of QEMU? Is it that folks want to manage the VM from both Vagrant as well as the UTM UI?

On my M1, I can cat ~/Library/Containers/com.utmapp.UTM/Data/Documents/ArchLinux\ ARM.utm/config.plist and see the configuration for the VM. Is Vagrant unable to store changes to this file if a user launched a VM from Vagrant?

If there's a problem reading/writing to config.plist, would it be easier to implement the QEMU provider?

BTW, UTM is awesome and was instrumental in me getting virtualization up and running on the M1, and it's inspired me to dig deeper and learn more lower in the stack. Thanks for implementing it and being helpful with all of my support questions.

[ppggff]

Try vagrant-qemu, a simple provider plugin using QEMU directly.

[scr-oath]

One of the reasons that I installed UTM is that the experience with X11 is pretty miserable if going through, say a Docker Desktop to XQuartz (the color tables are completely off and screen capturing doesn't get it right either), whereas UTM's display is quite nice (better than I remember VirtualBox being). I've tried out several linux variants and the Gnome Desktop and they're all very responsive (when virtualized), and any work with docker inside the VM can just mount the /tmp/.X11-unix dir for accessing the XServer as good as it gets through the unix socket.

Ultimately, I'd like to see proper support for UTM if the formats/configs can be negotiated/shared to make that so, but in the meantime, before I try it out, does the qemu provider have a UI facility at all? What does it use to display?

[jamesmortensen]

QEMU uses either a WebKitGTK window or VNC, to the best of my knowledge. You can check out the QEMU docs for the different display types you can pass to the -display argument.

I am not sure how good they are though. So far I have only used them on console only VMs.

[hftsai256]

One huge advantage to run vagrant on UTM instead of vanilla QEMU is vm.networking entitlement. Current select()/poll() from glib on macOS 11+ is broken and will corrupt external network packets (example).

Right now we either have to run vde-2 with vmnet as root, or run the entire qemu (ver 7.1+) as root with vmnet netdev. I wouldn't consider those solutions are secure and viable when scaled.

[Ileriayo]

I hope this gets implemented soon. 🤞

[rodrigc]

I don't know if this helps, but you may want to look at what Red Hat did with podman (a CLI which can run containers, a fully compatible yet complete replacement of Docker). On Mac:

podman machine init
podman machine start

will spin up a VM using qemu. This VM contains a Linux environment which runs all the containers started by podman .

The options they use to spin up the VM are here for amd64: https://github.com/containers/podman/blob/668deb2ff1cb918ec787e4c1de3a7c321b0def0b/pkg/machine/qemu/options_darwin_amd64.go

and here for arm64: https://github.com/containers/podman/blob/668deb2ff1cb918ec787e4c1de3a7c321b0def0b/pkg/machine/qemu/options_darwin_arm64.go

Maybe if Vagrant has a provider that just uses the bare minimum of qemu to spin up a VM, similar to how podman spins up a VM in qemu, that would be enough?

I think @baude and @jonpspri have worked on this.

[toddbu]

To add even more justification for UTM support... I use Vagrant for all my VM provisioning, but when it comes to port management then I do it ad-hoc through the UI so that I don't have to stop / start the VM. Like many others here, I'm transitioning from VirtualBox on an Intel Mac to UTM/QEMU on Apple Silicon because there is no VB solution. QEMU is a beast to deal with directly which is why UTM is my solution of choice. At this point if I'm forced to chose between UTM and Vagrant then I'll pick UTM. Most of my stuff runs in Docker anyway so it's not like VM setup is a huge hassle. I'm not eager to ditch Vagrant, but I get less operational hassle if I stick with UTM

[neonihil]

I'd love to see this happen! Do you guys need help on either side? I'm happy to throw in a few hours to see this happen. Feel free to DM me.

[tomjn]

@neonihil I believe that the open questions at the moment are from the UTM side, e.g. you could create a vagrant provider for UTM but knowing how to configure UTM to spin up a basic box without trigger user interface actions may be difficult.

Correct me please if I'm mistaken but:

• Some people have mentioned plists for configuration, a basic example configuration and an equivalent vagrant file that should map to each other would be helpful
• There may have been some hesitancy with the new vagrant-go and v2.3 release not being out, but this should now be resolved with the recent release
• I believe there is no machine/CLI/programmable UTM interface yet? But that one was planned? If this hasn't been done then this would be the primary blocker to a vagrant UTM provider, but if it has been done or it's in progress then collating that information would be valuable research for someone speccing out how a UTM provider would work. E.g. there are questions such as:
  • how would you turn the environment on and off? Destroy it? Create it?
  • given a libvirt box how would you give that to UTM?
  • how would you retrieve the status of the VM from UTM?

As for wether it will be officially supported rather than as a plugin, that's something Hashicorp would need to comment on, we can make it easier by helping with the technical details, but there's strong precedent for community written providers doing the work of adding support.

[ktprograms]

knowing how to configure UTM to spin up a basic box without trigger user interface actions may be difficult.

In UTM 4 (still in beta), there's support for removing all displays, so the VM runs headless. I'm not sure if the main UTM app will still show up, and as mentioned there isn't any CLI interface so you'd have to use the URL scheme to start VMs.

[therealmarv]

I don't want to derail the conversation about UTM but I want to mention that Tart VM for Apple Silicon supports now all needed technical requirements (Linux guests, API, shared folders etc.) for Vagrant with new macOS Ventura.

They just need help for creating the Vagrant provider for it:

https://github.com/hashicorp/vagrant/issues/12760#issuecomment-1228698988

[hftsai256]

Unfortunately Tart is out of the picture for me, because HyperKit does not support USB device passthorugh, and it is critical to embedded/firmware engineers to have an external debugger (mostly over USB) attached to the developing environment. I used to work with docker-machine but they deprecated the project last year, and therefore I am forced to work with UTM with physical USB device passthorugh.

[bernsel]

So many developers are using MacOS, and so many developers are using virtual machines. Many of those developers are now upgrading to Apple Silicon. What is Apple themselves recommending these developers to use? Many developers use for example Vargant + VirtualBox. You would think that Apple would have suggested an alternative for those who upgrade to M1/M2. Does anyone know if there is any kind of official recommendation or statement from Apple for these people?

[tomjn]

I don’t believe Apple has ever made such a recommendation arm or Intel. The closest you’ll get is their hypervisor APIs.

There are equivalents on Arm but only parallels has built a consumer facing vagrant provider plugin. You can’t just drop in a replacement vendor.

[osy]

There is now a scripting interface for UTM implementing the OSA for macOS apps: https://github.com/utmapp/UTM/blob/main/Scripting/UTM.sdef

We would like some feedback about any additional functionality that would make it easier to run Vagrant.

Note that there are already plans to implement some mechanism to talk with the QEMU Guest Agent although this is not currently implemented. That means right now there's no way to get any information out of a guest (such as SSH address).

[Cromian]

Would this be the interface/api a vagrant provider for UTM essentially interact with? (Sorry new to all of this but very interested to learn more to know how I can help).

[osy]

https://developer.apple.com/library/archive/documentation/AppleScript/Conceptual/AppleScriptX/Concepts/osa.html

It's an interface for other apps to interact with your app. I need to know what to provide in this interface for apps such as vagrant to function.

[tomjn]

I created a basic list of things that are needed for Tart integration in:

https://github.com/hashicorp/vagrant/issues/12760#issuecomment-1303559526

The same things/questions apply to UTM, I've been doing exploratory work with regards to a Tart provider, UTM support via AppleScript should be suitable, any Applescript APIs can be worked with via the osascript command.

At a minimum we would need to figure out what the applescript/osascript commands are to do these actions in UTM, much like hashicorp did for vboxmanage etc

[ktprograms]

Looking at the list, here's the things that seem to be needed on the UTM side:

• [ ] Getting VM IP address (will be possible with QEMU guest agent). After that SSH and running single commands should be possible.
• [ ] Setting up shared directories via apple events (I'm not sure if this is already possible).
• [ ] ID mapped mounts for 9pfs (Linux kernel side), this is needed since default UIDs on macOS don't match the 1000:1000 on Linux (it's 501:20 on macOS IIRC). I'm planning to work on this but not sure how easy it will be. If needed, the old WebDAV shares are still available I think. Alternatively, would SSHfs work on a macOS host without FUSE?

[goodwillcoding]

@bernsel

you would think that Apple would have suggested an alternative for those who upgrade to M1/M2.

Noone would think that, it's Apple .. they could not care less and you know it )))

[anarchy89]

Any progress on this ?

[tmm1]

@baude vagrant has a qemu provider? where?

[tmm1]

Who is "we"? And how is that relevant on this thread (which is about UTM)?

[tmm1]

Sorry I must be confused. I don't know how vagrant is related to podman.

I have a M1 Mac and I use vagrant. Can you tell me how I can hook it up to podman/qemu?

[baude]

Sorry I must be confused. I don't know how vagrant is related to podman.

I have a M1 Mac and I use vagrant. Can you tell me how I can hook it up to podman/qemu?

my apologies, i got my threads crossed. thanks @tmm1

[junosuarez]

@tmm1 you can run a vagrant machine backed by qemu on an M1/M2 mac via images like https://app.vagrantup.com/perk/boxes/ubuntu-2204-arm64 (see https://www.naut.ca/blog/2021/12/09/arm64-vm-on-macos-with-libvirt-qemu/ if you want to install an os from scratch). If you're interested in running network services only, this may suit your needs.

However, it's not a full solution. UTM handles certain edge cases better, such as sharing host USB devices. UTM does a great job integrating with the MacOS way that Apple wants you to do things - both in QEMU and Apple's own VirtualizationFramework. A downside is that UTM is very GUI-based. Thus, it would be valuable to get a declarative Vagrant workflow managing a UTM provider.

[osy]

As of v4.2.2 beta, we have added support for many new commands in the scripting interface including:

• Creating a new VM and duplicating an existing VM
• Configuration of an existing VM (including port forwarding on emulated VLAN and creating drives)
• Start/stop a VM
• Read/write files (when QEMU guest agent is installed) including support for reading/writing at offsets
• Executing scripts/commands (when QEMU guest agent is installed)

See the reference guide for more details: https://docs.getutm.app/scripting/reference/

Here is a cheat sheet with some example usages in Apple Script: https://docs.getutm.app/scripting/cheat-sheet/

In order to make a Vagrant plugin, you would probably need to use rb-appscript or RubyOSA. I don't have any experience with Ruby so YMMV. It would be great if someone can start working on a plugin and give feedback on any issues or other commands that they would need.

[bernsel]

We have found an alternative solution, which can replace VM Ware, so that we can still use Vagrant on M1/M2. The solution is summarized below.

• We download Docker Desktop (MacOS app).
• We create a Docker image for Ubuntu 22:04 (our chosen os for virtual machines).
• The Dockerfile for this image includes installing a vagrant user with the Vagrant public SSH Key.
• It also runs the container in privileged mode.
• We then define Docker as our VM provider in the Vagrantfile.

With this solution, we can run "vagrant up", "vagrant ssh" and other vagrant commands, as long as we first start Docker Desktop and have it running in the background. Docker replaces VM Ware.

[Fashaun]

@bernsel container solution is actually different with VM solution

[harlantc]

Using QEMU wouldn't work for us, as we want to virtualize macOS ARM on macOS ARM, and as far as I understand it UTM (and anyone else that wants to do this) needs to use Apple's own virtualization tools for this case, and QEMU for everything else.

Any chance this is still something being worked on? or would it be better to look at other tools such as tart?

[greenmoss]

Additional data point: I just tried installing Ubuntu 16.04 with UTM - manually download iso, use gui to configure, etc. I immediately hit an unrecoverable snag. Presumably with a Vagrant image/config any special tweaks would have already been resolved.

+1 for this issue.

[4WvgqJu]

So many developers are using MacOS, and so many developers are using virtual machines. Many of those developers are now upgrading to Apple Silicon. What is Apple themselves recommending these developers to use? Many developers use for example Vargant + VirtualBox. You would think that Apple would have suggested an alternative for those who upgrade to M1/M2. Does anyone know if there is any kind of official recommendation or statement from Apple for these people?

Good question! M2 VirtualBox combo isn't it. Did you tried UTM? I've tested it with Fedora 36 Workstation. It works, but it's Live! I've to install it complete first to script it with UTM Scripting. But's it's no possibility for provisioning to other users. I think that's why I support the issue here. But Apple is very, very restriktive in generally. I found a couple of apps didn't working anymore, until updating to something that Apple expects. So it will drive you crazy to keep it up to date. And VirtualBox it's Oracle's product. The borders walls against foreigns are going up high and more higher, everywhere for everything. I believe in peace and in constructive corporation. But now? Or what you think?

[tomjn]

Apple has never made a recommendation here be that for Intel or Arm, the choice to go with VirtualBox was made by Hashicorp.

UTM as a project doesn’t have the necessary APIs ready for a vagrant provider, you need to use the GUI to create VMs with it. CLI and machine APIs are planned but not complete. When they’re ready a provider will be possible. Nothing can be done from this end until then.

[4WvgqJu]

Apple has never made a recommendation here be that for Intel or Arm, the choice to go with VirtualBox was made by Hashicorp. UTM as a project doesn’t have the necessary APIs ready for a vagrant provider, you need to use the GUI to create VMs with it. CLI and machine APIs are planned but not complete. When they’re ready a provider will be possible. Nothing can be done from this end until then.

I agree with you, that's what I say! Apple isn't trusting Hashicorp, could someone think. Apple needs always a commercial product strategy, could someone think. Until it's experimental, all API's and Lines of Code and Documentation. I found documentation's attached examples one day and on next day already gone. Until, I will work and test UTM, maybe there are enough things I can test, each parameter from left to right, from smallest to biggest. Let's try and look! It's better like waiting I've lost all my hairs.

[osy]

Apple has never made a recommendation here be that for Intel or Arm, the choice to go with VirtualBox was made by Hashicorp. UTM as a project doesn’t have the necessary APIs ready for a vagrant provider, you need to use the GUI to create VMs with it. CLI and machine APIs are planned but not complete. When they’re ready a provider will be possible. Nothing can be done from this end until then.

UTM is an open source project and we've asked the community for suggestions on what APIs are the most needed and we've implemented many of them (file copying, USB mounting, etc). If there's any that is critical for Vagrant support that we currently don't support, we ask that you file an issue with the details.

Also there isn't anyone who is working on mapping the applescript interface with the Vagrant ruby based interface which would be a requirement. And I'm sure once someone starts working on that, they'll have better feedback on what's missing.

[4WvgqJu]

Apple has never made a recommendation here be that for Intel or Arm, the choice to go with VirtualBox was made by Hashicorp. UTM as a project doesn’t have the necessary APIs ready for a vagrant provider, you need to use the GUI to create VMs with it. CLI and machine APIs are planned but not complete. When they’re ready a provider will be possible. Nothing can be done from this end until then.

UTM is an open source project and we've asked the community for suggestions on what APIs are the most needed and we've implemented many of them (file copying, USB mounting, etc). If there's any that is critical for Vagrant support that we currently don't support, we ask that you file an issue with the details.

Also there isn't anyone who is working on mapping the applescript interface with the Vagrant ruby based interface which would be a requirement. And I'm sure once someone starts working on that, they'll have better feedback on what's missing.

Sorry, I'm subjective. I'm not a PO for UTM's Epics. Do you mean a Story would be enough? I'm new to M2, MacOS, UTM. I don't think I could write it down. From my perspective I need principal knowledge, reviews and time. Issue is open since 2021, we've 3Q/2023. 2 years past and no good news. But in generally it's easy:

• Start small, start simple, write always test when possible.
• Adapt the Vagrant API's requirements for UTM.
• Review the adaptions.
• Design the UTM cloud system architecture.
• Write adaption code.
• Test the implentations code.
• Find a provider to support the UTM Cloud.
• Configure images for different OS's locally, load images up into the UTM cloud.
• Create UTM provision scripts locally.
• Run the UTM provision scripts locally.
• Use the UTM CLI for workloads.
• Don't write documentation 'cause of TL;DR.

[osy]

Issue is open since 2021, we've 3Q/2023. 2 years past and no good news. But in generally it's easy

Please feel free to open a PR :)

[4WvgqJu]

Issue is open since 2021, we've 3Q/2023. 2 years past and no good news. But in generally it's easy

Please feel free to open a PR :)

Thank you for invitation writing the PR; Today I did a search and found a fork repo: https://github.com/vagrant-libvirt. What seemed to be, what I meaned. I will take a look on it later.

[tomjn]

Also there isn't anyone who is working on mapping the applescript interface with the Vagrant ruby based interface which would be a requirement. And I'm sure once someone starts working on that, they'll have better feedback on what's missing.

@osy A vagrant provider is a ruby plugin, to write one someone would need to know what the equivalent commands would be to control UTM. When the issue was opened there was no non-GUI method of creating and controlling VMs in UTM.

Additionally, modern vagrant is moving towards a golang approach making a ruby based provider a risky venture. It's also unlikely that Hashicorp will do the work of creating a UTM provider, or that it would be free if they did ( see VMWare ). I don't see any amount of support here changing that.

Creating a provider is a hefty undertaking, and would need to be done by the community, as someone who tried to build a Tart provider for Apple Silicon I've yet to see success, adding an additional layer of Applescript on top of plain CLI commands, the best UTM can provide is straight forward tracking of issues that affect us that are on their roadmap, and documentation of what's already been done. Last we heard a machine API, which was a massive blocker to attempting this, was planned but not the highest priority and there were no follow ups.

Looking at things, it seems that an Applescript API and a wrapper CLI utility was created since the issue was opened:

• https://docs.getutm.app/scripting/scripting/#command-line-interface
• https://docs.getutm.app/scripting/cheat-sheet/

Additionally, advice on how these should be packaged as box files for vagrant is necessary.

Now that the Applescript API is in place, the biggest thing that's needed for a UTM provider is volunteer programmers to write the UTM provider.

Thank you for invitation writing the PR; Today I did a search and found a fork repo: https://github.com/vagrant-libvirt. What seemed to be, what I meaned. I will take a look on it later.

@4WvgqJu I wouldn't do this alone, but if you plan to do it take a look at how the HyperV provider in vagrant itself is structured. It executes Powershell scripts rather than doing the logic in Ruby so that it can interact directly with Hyper-V APIs. Something similar but using Applescript would be useful. This does mean that a libvirt provider project isn't going to be any more helpful than the parallels or virtualbox providers though. Also look at what I'd done with https://github.com/tomjn/vagrant-tart-ruby, the inline docs and the WIP will help in understanding how providers and actions are structured

I'd also note that this will most likely need to be its own git repo, rather than a PR, and you'll need to discuss what the expectations are with UTM lest you have all of Apple Silicon vagrant support on your doorstep. Who at UTM can help with their API, who'll be responsible for support, will the repo migrate to their org, etc

[naveenrajm7]

Hello everyone!

I’m excited to share that I’ve developed a Vagrant plugin for UTM, which adds utm provider support for all Vagrant features. Beyond that it includes custom Vagrant disposable sessions using UTM’s disposable mode, and even preliminary support for Vagrant snapshots via qemu-img (Not available in UTM). Some features are currently built around the existing UTM API, and they will improve as more APIs are exposed in UTM.

I also have a working Packer plugin for UTM , which was used to build vagrant compatible boxes for the UTM provider.

You can find more details and try it out here: https://naveenrajm7.github.io/vagrant_utm/

I’d love to hear your feedback and see how it works for you.

Thanks!