QEMU builder with HVF accelerator support

[darkn3rd]

Please search the existing issues for relevant feature requests, and use the reaction feature (https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to add upvotes to pre-existing requests.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request. If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Description

Add HVF (intel and arm64) support for QEMU. The current version v1.1.0 according to the docs seems to only support KVM (ref)

Use Case(s)

There are a few images with qemu support on Vagrant Cloud, but it is not common, likely due to there being no support. Using the vagrant-qemu plugging, I have run systems using hvf with perk/ubuntu-2204-arm64 and also used the emulator q35 to run images for x86_64 with generic/ubuntu2204. but there seems to be no easy way to build the images.

Example from a tutorial:

cat <<EOF > Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "perk/ubuntu-2204-arm64"
  config.vm.provider "qemu" do |qe|
    qe.ssh_port = "50022" # change ssh port as needed
  end
end
EOF

vagrant up --provider=qemu
vagrant ssh

Potential configuration

Vagrant.configure("2") do |config|
  config.vm.box = "perk/ubuntu-2204-arm64"
  config.vm.provider "qemu" do |qe|
    qe.ssh_port = "50022"
  end

Potential References

[lbajolet-hashicorp]

Hi @darkn3rd,

I need some clarification, are you using Packer to build VM images? The samples you provided are for Vagrant only, so I wonder if your issue would not be a vagrant-qemu-provider instead?

Regarding KVM, if you are looking at the first sentence of the linked documentation, it's a bit misleading honestly. The qemu builder will create qemu disk images that you can then use to run VMs, the hypervisor in the background is irrelevant, anything that qemu supports we should too. Have you looked at the accelerator option? HVF is one of the documented options there in order to run your build process.

[darkn3rd]

The way the documentation is written, it seemed that only KVM accelerator was supported and HVF (both Intel and Apple Silicon) was not supported. So it would be nice to have an HVF solution path. I wanted Vagrant as an example consumer of the images, as there are not many easy paths for this, other than the plugin, or using xhyve, or maybe docker-machine (or minikube) with an xhyve or a mobykit driver. Docker Desktop uses HVF, but they use it as an embedded solution without support path to inject alternative images or for non-Docker-Desktop usages.

[lbajolet-hashicorp]

Hi @darkn3rd,

There is a documented option in the builder: accelerator, you can set this to hvf in order to run the machine with that accelerator underneath. Otherwise by default qemu if on Linux will use KVM, but on other OSes it will default to tcg. Maybe there's some heuristics we can use to default to using hvf on macOS, but in the meantime this is how it works.

Regarding the image itself, since it's a disk image (plus eventually some extra files like efivars if building an EFI image), it is not strictly bound to an hypervisor, you can build an image on KVM, and run it later on HVF (unless there's something I do not understand, and please feel free to correct me if that's not the case). If the fact that the image was built using tcg prevents vagrant from using HVF as the hypervisor for a qemu VM, this is likely a vagrant-qemu-provider issue, or maybe even a missing capability from Vagrant itself (I'm unfamiliar with the Vagrant code so I cannot help in pinpointing which component's missing something here), and in this case I suggest opening an issue to their repo.

In the meantime for this plugin, I suspect the culprit here is the The Qemu Packer builder is able to create [KVM](http://www.linux-kvm.org/) virtual machine images. intro sentence, which is kinda true (it defaults to KVM on a Linux environment) but also misleading as the image itself is not bound to an accelerator, which accelerator the machine runs on should be a runtime concern, not a build environment's.

I'll open a PR to change this sentence, but I'm not sure what else there is we can do here, if you have a suggestion I'd be happy to hear, and if you have the time, feel free to open a PR to update the documentation, we'll happily review!

[jacob-carlborg]

Otherwise by default qemu if on Linux will use KVM, but on other OSes it will default to tcg. Maybe there's some heuristics we can use to default to using hvf on macOS, but in the meantime this is how it works.

QEMU supports passing multiple accelerators and it will pick the most suitable. Perhaps delegate the responsibility to QEMU itself.

In the meantime for this plugin, I suspect the culprit here is the The Qemu Packer builder is able to create KVM virtual machine images. intro sentence, which is kinda true (it defaults to KVM on a Linux environment) but also misleading as the image itself is not bound to an accelerator, which accelerator the machine runs on should be a runtime concern, not a build environment's.

I'll open a PR to change this sentence, but I'm not sure what else there is we can do here, if you have a suggestion I'd be happy to hear, and if you have the time, feel free to open a PR to update the documentation, we'll happily review!

There seems to be a general problem and confusion (not specific to Packer) that the words "QEMU" and "KVM" are used interchangeable. QEMU is an emulator and a hypervisor/virtual machine monitor. KVM is a kernel level abstraction on top of CPU specific hypervisor features to allow hardware acceleration of running virtual machines. But I guess you already know all that. QEMU can use KVM, but it doesn't have to. A simple fix would be to replace "KVM" with "QEMU" in the documentation:

The Qemu Packer builder is able to create QEMU virtual machine images.

I guess it doesn't help with the confusion that on QEMU's web site it says: "Run KVM virtual machines".