Make Chapter 3 exercises M1/M2 compatible

jburns24 commented 1 year ago

The Problem

Chapter 2 of the bootcamp is centered around Virtualization. The chapter tells the following 'story' with its exercises. The camper is introduced to virtualization via a GUI and VirtualBox. Then they are introduced to Packer leveraging VirtualBox plugins to build custom vbox images. Then they are shown Vagrant and leverage their Packer script to create custom vbox machines with Vagrant.

As a company we have moved to Apple Silicon which is an ARM processor. VirtualBox is an x86 hypervisor and while there is a developer build that runs on m1/m2 chips this build essentially converts ARM commands to x86 commands and does so poorly. The result is this build does not work for any modern nix system. Threads suggest this will never really become a production ready thing and so alternatives are needed.

Options

The options for ARM virtualization is limited. Especially if we are looking for free solutions. There seems to be well supported paid solutions in VMWare and Parallels that integrate (seemingly) well with both Packer and Vagrant so either of these appear to be near 1-to-1 substitutions with VirtualBox bound exercises. As for free options there really is only QEMU.

QEMU

QEMU is a generic machine emulator with support for a wide array of architectures, aarch64 included. At a glance it seems like a good fit as there is the UTM project for a simple GUI experience, there is a Packer QEMU Builder, and a Vagrant QEMU Provider. But the devil is in the details.

UTM

This seems to work well enough for the exercises. It is already in the bootcamp and works with CentOS 9 images. I did have issues booting CentOS 7 images but if we have one (and the preferred most recent version of redhat we are probably OK)

QEMU Packer Builder

This builder targets KVM. This really is the 'normal' way that QEMU does virtualization. As previously stated QEMU is an emulator. Actually virtualization happens with KVM. KVM is and x86 project and not available on apple silicon. So without it QEMU has to emulate using accelerators and JIT translations. The packer builder is not really made for aarch64 versions of QEMU. And while it can can be modified I was unable to get it to work. Below are some of the stand out changes that had to be made and were I got blocked

source "qemu" "centos9" {
  iso_url = "https://mirror.stream.centos.org/9-stream/BaseOS/aarch64/iso/CentOS-Stream-9-20230508.0-aarch64-dvd1.iso"
  iso_checksum = "766b79db253487ad0ccebcc6dd4f72848357e10cffe03adc6a03a70d6bdbe6c7"
  disk_size = 10240
  headless = true
  use_default_display=true
  qemu_binary = "/opt/homebrew/bin/qemu-system-aarch64"
  machine_type = "virt"
  memory = 4096
  format = "qcow2"
  http_directory = "http"
  http_port_min = 8000
  http_port_max = 9000
  boot_wait = "2s"
  boot_command = ["<tab> text ks=http://{{ .HTTPIP }}:{{ .HTTPPort }}/anaconda-ks.cfg<enter><wait>"]
  ssh_username = "jburns"
  ssh_password = "vagrant"
  ssh_port = 22
  ssh_wait_timeout = "10000s"
  shutdown_command = "echo 'jburns'|sudo -S shutdown -P now"
}

build {
  sources = [
    "source.qemu.centos9"
  ]
}

headless = true -- was required as gtk is the default and did not work. I did not dig into this any further as headless should be fine accelerator='tcg' -- or omitting the accelerator parameter is required. This defaults to KVM which is x86 and does not work on m1/m2s. qemu_binary = "/opt/homebrew/bin/qemu-system-aarch64" -- This is required as by default since this targets KVM it tries to launch the x86 build of QEMU which will not work. Overriding this will target the correct arch for m1/m2 machine_type = "virt" -- Because we change the qemu_binary we have to update this option as the default does not work. This is concerning as it shows the plugin is opinionated towards x86 qemu and many options could need to be changed.

This is where my investigation was stopped. First this is hacky and not how this builder is intended to work. Secondly I never got it to work. With these changes I was running into this error 2023/05/15 14:14:23 packer-builder-qemu plugin: Qemu stderr: qemu-system-aarch64: no function defined to set boot device list for this architecture which from what I could gather was because the machine_type passed does not support the -boot flag that the plugin was trying to execute with. A suggested option was to supply the builder with etk2 bios but this needs to be compiled for you architecture and that is where I figured I am trying to fit a square peg into a round hole.

Vagrant QEMU plugin

Simple example of this worked OOTB and might be a good replacement. Though this is not a true test and we would need to see if we could get this provider to work with an image built by packer which I could never get working.

Parallels

At a glance it works on M1/M2 chips but is a paid product. Looks like we could get 5 seats for $89.99 a year with a 2 year contract. These seats can be reassigned. Parallels is well supported both in Packer and Vagrant. Could be an alternative but has a cost

VMWare

Also well supported with Packer and Vagrant and does work in M1/M2 chips. They do have a Free Personal License that we might be able to leverage for this section as it is for home use, educational purpose or Open Source contributors. Given the bootcamp is non commercial I am interested to see if this could work for us. Otherwise I think cost prohibits this option

jburns24 commented 1 year ago

[ ] Explore the personal use license for VMWare see if it works on m1 macs

jburns24 commented 8 months ago

This should be 'done' by https://github.com/liatrio/devops-bootcamp/pull/348 if the Finches run into an issue going through this chapter on arm processors we can make a new issue

liatrio / devops-bootcamp