Trellis and Apple Silicon

intelligence commented 3 years ago

[x] I've read the guidelines for Contributing to Roots Projects
[x] This request isn't a duplicate of an existing request
[x] This is not a personal support request that should be posted on the Roots Discourse community

Summary

Apple Silicon is here, VirtualBox support is far away (if ever) – what is the road going forward?

Motivation

Apple have started to ship their first arm based macs to customers and as many of us developers are sitting on macOS systems I'm curious if you've considered how the road forward will look? As I understand VirtualBox support seem far away, Docker is more probable. I love working with the Roots stack when doing WP work, would be nice to be able to do so in the future.

Some links with discussion around M1 based macs. https://forums.virtualbox.org/viewtopic.php?f=8&t=98742 https://www.docker.com/blog/apple-silicon-m1-chips-and-docker/ https://localwp.com/community/t/local-on-apple-silicon-macs/22834/14

swalkinshaw commented 2 years ago

I've finally published some docs on the Apple Silicon situation and Parallels: https://docs.roots.io/trellis/master/vagrant/

I'll keep this issue open for discussion and any updates.

mmoga-uf commented 2 years ago

Following the docs for Parallels with an M1, I get this error:

Stderr: Failed to start the VM: Unable to start "trellis-site.com" because your Mac is equipped with the Apple M1 chip that does not support Intel-based operating systems. To create a compatible virtual machine, use an ISO or VHDX image file with an ARM-based operating system.

I've installed the vagrant-parallels plugin and made the changes to the vagrant.default.yml file:

vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'

The only thing I really deviated from was the Vagrant version because I have 2.2.19 installed on my Mac. Has anyone else encountered this? I have Parallels trial version running.

mmoga-uf commented 2 years ago

Sorry, y'all! I had to delete the .vagrant folder and try again. Going to backdate Vagrant now...

fowlercraig commented 2 years ago

@mmoga-uf how are you backdating Vagrant? I've been attempting with Brew, but haven't had any luck.

swalkinshaw commented 2 years ago

See https://discourse.roots.io/t/downgrading-vagrant-version/21742/4

mmoga-uf commented 2 years ago

@mmoga-uf how are you backdating Vagrant? I've been attempting with Brew, but haven't had any luck.

I may have been missing some important point, but I actually kept Vagrant 2.2.19 and then made this change https://github.com/hashicorp/vagrant/issues/12583#issuecomment-985787134.

I got Trellis/Bedrock/Sage all working locally, but then had to bail before doing anything remotely because I work at a RHEL LAMP institution. 😥

techieshark commented 2 years ago

This may be a bit far afield but…

I wonder if the virtualization changes announced at WWDC 2022 will pave the way for some new alternatives.

Here's an Apple demo video showing how some Swift code can be used to set up a Linux VM (jump to 15:13 to see the setup of Linux on Mac). Here's something similar but in writing.

From this blog post:

In macOS Ventura, Apple is introducing support for the EFI boot loader, which is able to discover any virtual device attached to a VM from which to boot. This makes it possible to download an ISO Linux image, attach it to a virtual storage device, create a boot loader, and boot into the OS, as Poulin shows. This is possible both on Intel and ARM CPUs provided you download the appropriate distribution for your hardware.

Additionally, with macOS Ventura, you will be able to use VirtioGPU 2D, a virtualization device that allows Linux to provide interfaces to the host. This makes it possible to display a Linux GUI inside a macOS window, similarly to how you can display a macOS GUI within a macOS window.

Possibly the most interesting new virtualization feature coming with macOS Ventura is support for Rosetta to run unmodified x86-64 Linux binaries.

(I'm also interested in the open source options… @mikaelwedemeyer I wonder if you had any luck with multipass from Canonical?)

swalkinshaw commented 2 years ago

I briefly read about those macOS Ventura changes too but honestly was a bit confused. So it might improve compatibility for these cases, but performance would still suffer since it's under emulation (unless I'm mistaken). So ultimately still running ARM-based OS's would give the best performance.

techieshark commented 2 years ago

@swalkinshaw it sounds like the performance impact might not be so bad… I'm only just reading up on it now but it seems like to some extent they're able translate the machine instructions and store the translation, so the second run is faster. Also my understanding is they're offloading some or all of that to the GPU.

With new Apple Silicon machines, this is done by Rosetta 2. When you try to open an Intel binary on an M1-based machine, MacOS hands it off to a program called Rosetta 2. Rosetta 2 does an on-the-fly translation of the x86 code into Apple Silicon code, saves it, and then runs the translated code. Some elements are emulated while other elements are completely transcoded. The first time you run an Intel-based program, it may take a little while for the program to start running. That's because Rosetta is doing a translation pass first. Subsequent runs will then be faster because the translation has already been done. (ref)

One major concern with Rosetta is performance. With the transition from PPC to x86, one factor slowing down Rosetta was the different byte ordering used by the two platforms, with PowerPC being a big-endian architecture, and x86 little-endian. While byte ordering is not a problem for the transition from x86 to ARM, another issue related to memory, namely the memory consistency model total store ordering (TSO), could hamper performance in this case. To prevent this from happening, Apple added support for x86 memory ordering to the M1 CPU, as Robert Graham noted on Twitter. (ref)

But yeah I haven't seen much of a technical analysis for Ventura so … I suppose safest bet for now is to use ARM-based OS like you say.

BTW I have mad some progress with the Trellis + Vagrant + Docker setup using the tips above (which uses ARM-based Linux) - https://github.com/roots/trellis/issues/1253#issuecomment-974896550

jgarib commented 2 years ago

Today, while spinning up a new project I had to change the vagrant default to:

vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64' vagrant_box_version: '1.0.0'

The 1.0.1 release of the arm box wasn't booting so removing the >= from the version definition got me going. I'll look into it more later, unless anyone here has already run into similar issue and wants to chime in.

craigpearson commented 2 years ago

Yeah there's an issue with the new vagrant box version, and only version 1.0.0 works without issue

It seems to be network related, and I assume it's either due to the creator exporting the VM without configuring a new MAC address for virtual network cards or related to some of the other changes with parallel tools which are included in v1.0.1

I think this only becomes problematic when you have multiple boxes, due to the mac addresses of the boxes being the same this seems to break parallels due to the same DHCP address being assigned on the required parallels shared network.

Once you hit this issue you have to destroy all your machines, do a hard system restart and rebuild - so it's a bit of a pain

Will see if I can raise with the creator

jgarib commented 2 years ago

Good info. Thanks! Thankfully all my previous boxes (at least the three I've run since) are OK and avoided any collateral damage... at least so far.

techieshark commented 2 years ago

@j-funk you noted the networking issue with Docker - https://github.com/roots/trellis/issues/1253#issuecomment-974896550

After hitting that myself and digging in a bit, it seems like it might be due to the firewall settings Trellis includes (via ferm). I seem to be having a little luck working around it by adding nameserver 1.1.1.1 to the /etc/resolv.conf on the docker box ; see here for how I persisted this change via a new little Ansible role - https://discourse.roots.io/t/trellis-provisioning-breaks-dns-in-docker/23808/2?u=techieshark

Bozzified commented 2 years ago

Yeah there's an issue with the new vagrant box version, and only version 1.0.0 works without issue

It seems to be network related, and I assume it's either due to the creator exporting the VM without configuring a new MAC address for virtual network cards or related to some of the other changes with parallel tools which are included in v1.0.1

I think this only becomes problematic when you have multiple boxes, due to the mac addresses of the boxes being the same this seems to break parallels due to the same DHCP address being assigned on the required parallels shared network.

Once you hit this issue you have to destroy all your machines, do a hard system restart and rebuild - so it's a bit of a pain

Will see if I can raise with the creator

I'm glad I wasn't going crazy. I noticed this happening and it is huge pain for me. I have multiple boxes running for multiple sites and this has been giving me nightmares.

Hopefully it gets fixed.

swalkinshaw commented 1 year ago

While Trellis has supported Apple Silicon for a while now, it still required two manual changes (Vagrant box + MailHog). This is long overdue, but I'm happy to say that both those issues are fixed and there's no more manual changes needed. Apple Silicon should just work out of the box (via a Vagrant provider that supports ARM of course).

Thanks for everyone's feedback and help 🚀

jkananen commented 1 year ago

@swalkinshaw it seems trellis new misses these last two updates.

As trellis up failed, I compared my trellis files against trellis master, and clearly #1431 and #1432 weren't in. I've patched my workarea and all is now well.

swalkinshaw commented 1 year ago

Ah, yeah apologies @jkananen I need to do a new Trellis release which I'll do when I'm back from vacation.

In the meantime you can always use trellis new --trellis-version=dev to get the version from master.

nikitasol commented 1 year ago

Anyone had a chance to try Virtualbox ARM version yet? -> https://download.virtualbox.org/virtualbox/7.0.4/VirtualBox-7.0.4_BETA4-154605-macOSArm64.dmg

tomdevisser commented 1 year ago

Anyone had a chance to try Virtualbox ARM version yet? -> https://download.virtualbox.org/virtualbox/7.0.4/VirtualBox-7.0.4_BETA4-154605-macOSArm64.dmg

Doesn't work yet.


The box you're attempting to add doesn't support the provider
you requested. Please find an alternate box or use an alternate
provider. Double-check your requested provider to verify you didn't
simply misspell it.

If you're adding a box from HashiCorp's Vagrant Cloud, make sure the box is
released.

Name: bento/ubuntu-22.04-arm64
Address: https://vagrantcloud.com/bento/ubuntu-22.04-arm64
Requested provider: [:virtualbox]```

swalkinshaw commented 1 year ago

We just released VM integration into trellis-cli powered by Lima 🎉 See https://roots.io/introducing-lima-to-trellis-for-faster-local-development/ for more details and https://github.com/roots/trellis-cli/pull/346 for some documentation.

This has many benefits over Vagrant + VirtualBox/Parallels:

it's free and open source
it lighter weight and simpler
commands will run much faster since Vagrant is slow
VMs should run faster as well since it uses macOS virtualization framework which is as close to native speed as you can get

stevedsross commented 1 year ago

This is awesome. Thanks for sharing. I'll start testing this in the next couple of weeks.

minemindmedia commented 1 year ago

I am trying to follow along here and get some projects booted up on my new m2 macbook but run into this error which is not very detailed and I've hit a wall. Any ideas how to get passed this?

==> default: Booting VM...

There was an error while command execution. The command and stderr is shown below.

Command: ["/usr/local/bin/prlctl", "start", "03376801-9a15-421d-a445-ef786b52765a"]

Stderr: Failed to start the VM: The virtual machine cannot be started due to a critical error.

I've updated my vagrantfile as suggested and have destroyed and attempted to launch over and over with no success.

vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'

roots / trellis

Trellis and Apple Silicon #1253

Summary

Motivation