Closed intelligence closed 1 year ago
I haven't looked into this enough yet, but my interpretation of the current state of things is:
I think the key point is that Trellis is coupled to x86 more so than VirtualBox itself. Trellis already supports other Vagrant providers which some people use.
There's two biggest questions:
I personally don't really know the implications of running difference architectures in dev/prod yet. So even if Trellis found a way to migrate to Docker... there's still a lot of potential problems and big open questions.
My stance for now is nothing will change for a while. Anyone running M1 chips is unfortunately out of luck running for Trellis for at least the short-term and almost certainly for the medium-term too.
1 sounds a bit farther away, but 2 looks like a necessity already for anyone with ARM macs.
@swalkinshaw Scott, as for short-term, do you have weeks or rather months in mind?
If you want to run ARM in development now then I'd try and find another Vagrant provider that supports it. VirtualBox isn't the only choice so Trellis doesn't really need to do anything to support that (as far as I know).
A bit over my head, but I would like to try it. Would you suggest replacing virtualbox with docker? I remember seeing a couple of years ago a docker feed on discourse, but not sure if it gave any solid fruits in the end.
From what I can see, there is no working provider at the moment. Docker is on standby waiting for upstream updates from Go and others with no ETA, Parallels is still in development with no ETA either, VMware is committed as well, but completely silent about roadmap. So there is no possibility to try out the 2nd option for the time being. But it would be wise to at least leave a note for any silicon users stating that the Trellis framework is not compatible with Apple M1 laptops until at least some of the Vagrant providers start rolling out arm tailored versions.
Ah, yeah sorry I wasn't sure what the status of Parallels and VMware was yet. I had just read they had betas or were working on it.
But yes we definitely need to document this limitation somewhere.
Seems like great progress is being made!
https://www.xda-developers.com/apple-silicon-mac-boot-windows-10-and-linux-virtualization/
So developers who bought new Macbook Pro with M1 are currently not supported by Trellis? Docker seems to be working on M1, but Trellis is not compatible with Docker. Maby Trellis should really consider Docker as other provider than VirtualBox?
@Pls I think Docker is a bit different solution to the question as it basically replaces trellis instead of just creating a vm. But you can try out docker approach. You can find more information here: https://roots.io/docs/bedrock/master/local-development/#additional-resources.
I thought about Rosetta 2 since it allows some apps not made for the M1 chip to actually run, but unfortunately they don't support virtual machines.
What can’t Rosetta 2 translate? Rosetta cannot translate kernel extensions or Virtual Machine apps that virtualize x86_64 computer platforms. Developers should be aware that Rosetta is also unable to translate AVX, AVX2, and AVX512 vector instructions.
Leaving it as a reference for anyone stumbling here.
Just got a new iMac with the M1 chip, would love to know what the gameplan is for making it compatible with the Roots suite. Is it just a waiting game at this point?
Has anyone attempted Trellis on ARM in development on the M1? Parallels should support that. https://www.parallels.com/blogs/parallels-desktop-m1/
Can we get support for m1 macs through docker? It looks like virtualbox and even vmware are dragging their feet on support the macs and some people got vagrant working on docker: https://dev.to/taybenlor/running-vagrant-on-an-m1-apple-silicon-using-docker-3fh4
It would be great if we could flip a flag and spin up a docker image instead of a virtualbox machine, not sure how hard this would be in practice though.
-- @Digital-Nomad https://github.com/roots/trellis/issues/1302#issue-930245243
I've been looking into this a bit more lately, and here's what I've found.
The new M1 Pro and M1 Max CPUs don't change anything virtualization wise unfortunately. They are the same as the original M1 in that regard.
There has been absolutely no movement on Virtualbox offering ARM support and I highly doubt there ever will be. It defeats the purpose of VirtualBox which is very tied, by design, to x86.
While Trellis defaults to VirtualBox (since it's open source and free), it's always supported other providers like VMWare and Parallels.
As mentioned above, Parallels does have support for M1 macs but only for ARM based operating systems.
Likewise, VMware Fusion now supports ARM-based Linux distributions just like Parallels.
One issue with all of these is integrating them with Vagrant. docker has by far the most mature Vagrant integration with a 1st-party provider. Lima and UTM have none so far. vagrant-libvirt
exists but has a lot of issues on macOS
The best bet at this point is running Virtualbox on Intel Macs and some other ARM-based virtualization solution on M1 chips. This isn't 100% ideal since it's likely your production server will be running x86 for the foreseeable future; however, I'm not sure it matters in reality. While there are difference between x86 and ARM, it likely does not matter for the purposes of running a web server with WordPress. PHP should be PHP; Nginx should be Nginx, etc.
It's a small trade-off we'll have to make and be okay with in the end.
This space is changing every day and it's hard to keep track. There's a lot of solutions which are like 50-80% of the way there, but not 100% unfortunately. Right now the most promising solution is:
Of course Docker's default design is not compatible with Trellis because docker images/containers don't like to run multiple services and lack an initd system. This has previously been discussed, but I did find a useful base docker image which solves most of these problems: https://medium.com/nerd-for-tech/developing-on-apple-m1-silicon-with-virtual-environments-4f5f0765fd2f
I've been testing this out on an x86 Mac, and I've run into a few problems (NFS and networking related), but are hopefully solvable.
If anyone wants to help test out any of the other Vagrant providers or VM solutions, please let me know or post your results. It's also helpful for people to post updates about these various projects if they come across them.
Further reading:
If you have an M1 Mac and have Parallels or VMware, you can help by trying out an ARM based Ubuntu box.
Since Trellis already supports Parallels and VMware fusion, it's possible to try using an ARM-based Ubuntu vagrant image and get Trellis working "as is".
The main Ubuntu vagrant box we use is bento/ubuntu-20-04
which doesn't yet have support for ARM. However, https://github.com/chef/bento/pull/1374 looks like it has most of it ready. I think you'd need to packer
to actually create that box/image and use it locally.
Ok! I got this to work with parallels on an M1 Pro macbook pro. I'm not the smartest person here so I'm going to lay out some steps in hopes of helping others like me. The machine provisions without errors, but I have not extensively tested beyond this.
**note I believe parallels pro is needed which is what my trial is for this test. The basic version does not provide the cli tools. https://www.parallels.com/products/desktop/pro/#compare
Following swalkinshaw's extensive legwork, using the chef/bento vagrant box on parallels is the only one I could get working. I tried using VMware fusion but I think more work needs to be done to get that working. Jeffnoxon already has a vagrant box compatible with parallels up on the vagrant cloud (https://app.vagrantup.com/jeffnoxon/boxes/ubuntu-20.04-arm64). I'm not sure if it's safe to just reference this box or if something more formal needs to be done.
I added that box to the vagrant.default.yml
vagrant_box: 'jeffnoxon/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'
I also needed to make a slight change in the maria-db role. roles > mariadb > defaults > main.yml -- change the arch from amd64 to arm64
mariadb_ppa: "deb [arch=arm64] http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.5/ubuntu {{ ansible_distribution_release }} main"
after that, vagrant up using the parallels provider (make sure you install the parallels vagrant provider plugin (https://parallels.github.io/vagrant-parallels/docs/installation/)
vagrant up --provider=parallels
And success! https://prnt.sc/1xujta1. Loading the site at example.test works and everything appears good. Obviously this will only work for a development environment. I have not tested provisioning a staging / production server on arm64... one step at a time. I can now at least continue developing in glorious real all day battery land and I'll just copy the files to an x86 based machine for the rest of the workflow until I can get more experience with the rest of it.
I'll follow up here with any issues as I find them.
@jgarib awesome, thank you so much! I'm not that surprised that this "just" worked. Ubuntu should be Ubuntu, so it makes sense it worked once you got that up and running.
re the jeffnoxon/ubuntu-20.04-arm64
box: the base Vagrant box shouldn't matter that much, as long as it's a standard Ubuntu image 👍
I'll grep the source again, but assuming MariaDB is the only place with a hardcoded architecture, we can use Ansible's ansible_architecture
to support both (it just needs a mapping).
@jgarib did Nginx not fail with the same issue? It has deb [arch=amd64]
. Either way, I'll fix both 👍
No issue with nginx, though you're right.. you'd think it would fail the same as mariadb. Here is the output of the nginx section of the provisioning with arch=amd64 still set. I can run a quick test with the same change to verify it still works if you'd like?
I'm pretty sure we can just remove the deb arch options entirely. It defaults to the arch that dpkg
supports. Running dpkg --print-architecture
on my x86 VM just outputs amd64
as you'd expect. I'm assuming that if you run it you'll get back arm64
. This makes sense to me since you shouldn't have to explicitly tell apt
what your architecture is; it should just default to using your current arch.
Sorry this is all on the virtual machine side (in Ubuntu)
This is working on the x86 side for me and I'm assuming it will work for M1s. https://github.com/roots/trellis/pull/1318
Confirmed working on ARM side. So once that is merged the only change needed to use trellis in a development environment on an M1 mac is:
jeffnoxon/ubuntu-20.04-arm64
) in vagrant.default.yml or whatever change you make to support both!Thanks much Scott!
🎉 yep, but I can probably update the Vagrantfile
to use that box automatically as well so ideally there's no manual steps needed.
Just chiming in to say that I've got Trellis working on an M1 Mac Mini by following these instructions with Parallels Pro.
Is it possible to set a default provider in the vagrant.default.yml file or anywhere else? It's not ideal to have to type the provider flag every time I provision a machine, though not the end of the world.
Trellis’s Vagrantfile tries to auto select a provider in this order:
if multiple providers are installed, first one will be used.
Alternatively, you can overrides it via the VAGRANT_DEFAULT_PROVIDER
environmental variable.
See: https://www.vagrantup.com/docs/providers/basic_usage#default-provider
Yeah if you only have Parallels installed it should be the default? Is that not what's happening?
If you have an M1 mac then you definitely shouldn't have VirtualBox installed since it won't work anyway.
That was it. I had VirtualBox installed somehow from when i was first getting started on this M1 machine.
Also posting here to say that this discussion led to a resolution for me. I'm running Parallels on a trial right now and all systems go. I moved Parallels entry up in Vagrantfile
so I wouldn't have to specify a provider. I don't have VirtualBox installed but it was still defaulting to it.
I'm running the full stack (Trellis/Bedrock/Sage) so I'd also like to reference the discussion on Sage's repo resolving an issue with Node-Sass while building with newer versions of Nodejs since previous versions aren't compatible with Apple Silicon. https://github.com/roots/sage/issues/2411
moved Parallels entry up in Vagrantfile so I wouldn't have to specify a provider. I don't have VirtualBox installed but it was still defaulting to it.
What is the error message when you vagrant up
without --provider
and without re-ordering Vagrantfile
?
(Please test it on a brand new VM because vagrant "remembers" which provider was used on first vagrant up
)
What is the error message when you
vagrant up
without--provider
and without re-orderingVagrantfile
?
Freshly cloned trellis folder with the latest changes pulled in, spinning up a brand new virtual machine.
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Box 'jeffnoxon/ubuntu-20.04-arm64' could not be found. Attempting to find and install...
default: Box Provider: virtualbox
default: Box Version: >= 1.0.0
==> default: Loading metadata for box 'jeffnoxon/ubuntu-20.04-arm64'
default: URL: https://vagrantcloud.com/jeffnoxon/ubuntu-20.04-arm64
The box you're attempting to add doesn't support the provider
you requested. Please find an alternate box or use an alternate
provider. Double-check your requested provider to verify you didn't
simply misspell it.
If you're adding a box from HashiCorp's Vagrant Cloud, make sure the box is
released.
Name: jeffnoxon/ubuntu-20.04-arm64
Address: https://vagrantcloud.com/jeffnoxon/ubuntu-20.04-arm64
Requested provider: [:virtualbox]
So local development is possible. Any thoughts on deployments? Is it possible to setup and deploy to staging/production without any hiccups?
Deploying with Trellis/Ansible should work fine since you're just running Python locally. The only real issue in development was the virtual machine part, not Trellis itself.
The above solution with parallels pro was working for me last week. However, I'm once again encountering issues when creating a new project and firing up.
Failing at the following.
TASK [mariadb : Install MySQL server] ******************************************
System info:
Ansible 2.9.10; Vagrant 2.2.14; Linux
Trellis 1.9.0: October 27th, 2021
---------------------------------------------------
Failed to update apt cache: W:Updating from such a repository can't be done
securely, and is therefore disabled by default., W:See apt-secure(8) manpage
for repository creation and user configuration details., E:The repository
'http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.5/ubuntu focal Release'
no longer has a Release file.
fatal: [default]: FAILED! => {"changed": false}
Inside my vagrant.default.yml my box is set to 'jeffnoxon/ubuntu-20.04-arm64' Inside mariadb/defaults/main.yml mariadb_ppa is set to "deb http://nyc2.mirrors.digitalocean.com/mariadb/repo/10.5/ubuntu {{ ansible_distribution_release }} main"
Anyone else encountering this issue on an m1 mac?
Digitalocean messed up their mirror: https://twitter.com/tarendai/status/1458222036657086465?s=21
Try using another one: https://mirrors.gigenet.com/mariadb/repo/10.5/ubuntu
You can override this var: https://github.com/roots/trellis/blob/master/roles/mariadb/defaults/main.yml#L3
Yep. This ^ was it.
The Mirror link worked like a charm. Thanks for the assist.
Have been following this very closely, but am not in love with closed-source options on the virtualization side.
Multipass is a vm solution from canonical just for ubuntu images, it now has M1 support: https://multipass.run/ There is also a Vagrant provider: https://github.com/Fred78290/vagrant-multipass
Was going to investigate if this would be a viable solution, interested if anyone has seen it, or if there are issues that make it a non-starter.
It looks like there is some movement going on with VB https://www.virtualbox.org/changeset/92220/vbox
Got this running on my M1 MBP with Parallels Pro, thanks everyone!
I'm not a fan of the closed source parallels
either. I was able to get part of the way there using Docker for Mac and the config from this Medium blog post.
I made the following changes to Vagrantfile
:
# Provider for Docker
config.vm.provider :docker do |docker, override|
override.vm.box = nil
docker.image = "rofrano/vagrant-provider:ubuntu"
docker.remains_running = true
docker.has_ssh = true
docker.privileged = true
docker.volumes = ["/sys/fs/cgroup:/sys/fs/cgroup:ro"]
end
And to vagrant.default.yml
:
vagrant_mount_type: 'docker'
vagrant_box: 'rofrano/vagrant-provider:ubuntu'
This was enough to get Trellis to create a machine, but I get some networking errors, perhaps someone else can take it form here, since I am out of ideas:
... Could not resolve 'ports.ubuntu.com'"]}
RUNNING HANDLER [restart fail2ban] *********************************************
changed: [default]
RUNNING HANDLER [restart ferm] *************************************************
skipping: [default]
PLAY RECAP *********************************************************************
default : ok=27 changed=11 unreachable=0 failed=1 skipped=15 rescued=0 ignored=0
I made it kinda-sorta work with VMWare Fusion Tech Preview (direct link) and this box:
vagrant_box: 'rkrause/ubuntu-20.04-arm64'
vagrant_box_version: '>= 1.0.0'
I'm saying kinda-sorta because:
--http1.1
or just default curl either returns empty response from server
or `curl: (92) HTTP/2 stream 1 was not closed cleanly before end of the underlying stream~ See EDIT 3curl -vk https://what.test/index.html
* Trying 192.168.138.128:443...
* Connected to what.test (192.168.138.128) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
* CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=what.test
* start date: Nov 23 17:07:37 2021 GMT
* expire date: Feb 26 17:07:37 2024 GMT
* issuer: CN=what.test
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x131012400)
> GET /index.html HTTP/2
> Host: what.test
> user-agent: curl/7.77.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* HTTP/2 stream 1 was not closed cleanly before end of the underlying stream
* stopped the pause stream!
* Connection #0 to host what.test left intact
curl: (92) HTTP/2 stream 1 was not closed cleanly before end of the underlying stream
curl -vk http://what.test/index.html
* Trying 192.168.138.128:80...
* Connected to what.test (192.168.138.128) port 80 (#0)
> GET /index.html HTTP/1.1
> Host: what.test
> User-Agent: curl/7.77.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 24 Nov 2021 02:44:29 GMT
< Content-Type: text/html
< Content-Length: 5
< Last-Modified: Tue, 23 Nov 2021 17:58:10 GMT
< Connection: keep-alive
< ETag: "619d2bb2-5"
< Accept-Ranges: bytes
<
WHAT
* Connection #0 to host what.test left intact
I also tried Parallels and it also kinda-sorta worked but since it's a paid subscription I decided to try VMWare Fusion.
[EDIT] SSL works just fine from inside the VM I can do:
root@what:/srv/www/what.test/current/web# curl -k https://what.test/index.html
WHAT
[EDIT 2]
Just tested from the browser and it works 🤯
[EDIT 3]
As it turns out, workers kept dying with
worker process 7569 exited on signal 11 (core dumped)
I started hacking off parts of config until I found the culprit in h5bp/directives-only/ssl.conf
which came from the old version of Trellis (circa 2019) that contained ssl_prefer_server_ciphers on;
, I swapped that config with what current Trellis has and it started working.
PHEW what a ride!
Thanks @rinatkhaziev – after my Parallels trial expired, following your instructions, I managed to get trellis working with the VMWare fusion preview using the rkrause/ubuntu-20.04-arm64
box.
@rinatkhaziev was there anything else you used, like plugins or any other settings? I'm getting errors when I try to use the VMWare Fusion Preview you linked.
For anyone else who runs into mount issues with Parallels, some of the folders were failing with this error:
Invalid mount options: ["dmode=755", "fmode=644"]
I found this article and ended up modifying any line in my Vagrantfile
that looked contains mount_options: mount_options(vagrant_mount_type, dmode: 755, fmode: 755)
to mount_options: ["share"]
Hope this helps someone.
@j-funk I haven't touched anything else besides mailhog dists which officially are not compiled for ARM64 by setting
mailhog_binary_url: "https://github.com/evertiro/MailHog/releases/download/v1.0.1-M1/MailHog_linux_arm64"
mhsendmail_binary_url: "https://github.com/evertiro/mhsendmail/releases/download/v0.2.0-M1/mhsendmail_linux_arm64"
Here's how the mounts part looks for me (mount type is set to nfs)
if vagrant_mount_type != 'nfs' || Vagrant::Util::Platform.wsl? || (Vagrant::Util::Platform.windows? && !Vagrant.has_plugin?('vagrant-winnfsd'))
vagrant_mount_type = nil if vagrant_mount_type == 'nfs'
trellis_config.wordpress_sites.each_pair do |name, site|
config.vm.synced_folder local_site_path(site), remote_site_path(name, site), owner: 'vagrant', group: 'www-data', mount_options: mount_options(vagrant_mount_type, dmode: 776, fmode: 775), type: vagrant_mount_type, **extra_options
end
config.vm.synced_folder ANSIBLE_PATH, ANSIBLE_PATH_ON_VM, mount_options: mount_options(vagrant_mount_type, dmode: 755, fmode: 644), type: vagrant_mount_type, **extra_options
config.vm.synced_folder File.join(ANSIBLE_PATH, 'bin'), bin_path, mount_options: mount_options(vagrant_mount_type, dmode: 755, fmode: 755), type: vagrant_mount_type, **extra_options
elsif !Vagrant.has_plugin?('vagrant-bindfs')
fail_with_message "vagrant-bindfs missing, please install the plugin with this command:\nvagrant plugin install vagrant-bindfs"
else
trellis_config.wordpress_sites.each_pair do |name, site|
config.vm.synced_folder local_site_path(site), nfs_path(name), type: 'nfs'
config.bindfs.bind_folder nfs_path(name), remote_site_path(name, site), u: 'vagrant', g: 'www-data', o: 'nonempty'
end
config.vm.synced_folder ANSIBLE_PATH, '/ansible-nfs', type: 'nfs'
config.bindfs.bind_folder '/ansible-nfs', ANSIBLE_PATH_ON_VM, o: 'nonempty', p: '0644,a+D'
config.bindfs.bind_folder bin_path, bin_path, perms: '0755'
end
vconfig.fetch('vagrant_synced_folders', []).each do |folder|
options = {
type: folder.fetch('type', 'nfs'),
create: folder.fetch('create', false),
mount_options: folder.fetch('mount_options', [])
}
destination_folder = folder.fetch('bindfs', true) ? nfs_path(folder['destination']) : folder['destination']
config.vm.synced_folder folder['local_path'], destination_folder, options
if folder.fetch('bindfs', true)
config.bindfs.bind_folder destination_folder, folder['destination'], folder.fetch('bindfs_options', {})
end
end
I finally have an M1 Mac to test Trellis on! Quick notes:
The main known issue that's remaining is with mailhog as @rinatkhaziev posted above. Updating the package URLs works, but there might be something we can build into Trellis.
Same goes with the Vagrant box. I attempted this in https://github.com/roots/trellis/pull/1346 but due to Rosetta emulation it's not that easy. uname -m
actually reports x86_64
from within Vagrant since it's run in emulation mode. I found a way to reliably detect ARM based Macs now.
@swalkinshaw
I also tried the VMWare Fusion preview (which is free right now) and had nothing but trouble
Out of curiosity, what did you encounter?
I actually abandoned it after that last issue and got Parallels working fine 🤷♂️
After trying both, I ended up shelling out for Parallels for a year. There were too many issues with VMWare, namely:
Should have published this sooner, but Roots joined the Parallels affiliate program. So if people do have to purchase Parallels, you can support us by doing it through this link: https://prf.hn/l/KzkNLZB
Note: just make sure to select your proper country/currency.
Summary
Apple Silicon is here, VirtualBox support is far away (if ever) – what is the road going forward?
Motivation
Apple have started to ship their first arm based macs to customers and as many of us developers are sitting on macOS systems I'm curious if you've considered how the road forward will look? As I understand VirtualBox support seem far away, Docker is more probable. I love working with the Roots stack when doing WP work, would be nice to be able to do so in the future.
Some links with discussion around M1 based macs. https://forums.virtualbox.org/viewtopic.php?f=8&t=98742 https://www.docker.com/blog/apple-silicon-m1-chips-and-docker/ https://localwp.com/community/t/local-on-apple-silicon-macs/22834/14