search

uselagoon / lagoon

Lagoon, the developer-focused application delivery platform
https://docs.lagoon.sh/
Apache License 2.0
568 stars 148 forks source link

Request for cross built linux/arm64 versions of Lagoon containers #2521

Closed ocean closed 2 years ago

ocean commented 3 years ago

I'm on a new "Apple Silicon" M1 MacBook Pro, which in theory can run amd64 containers in emulation, but it doesn't seem to work. linux/arm64 containers run fine, and there are already a lot of linux/arm64 containers on Docker Hub.

As more and more developers move to the new Apple Silicon, this will affect more and more people.

Docker "buildx" command allows for cross-platform building, I'll add some more links to this issue with some examples.

If the Amazee Lagoon container build repository is open source, I'd be very happy to fork and contribute, but I can't find it linked from Docker Hub.

tobybellwood commented 3 years ago

Hi @ocean - thanks for being keen! We've only taken a cursory glance at ARM at this stage. We do intend to pursue multi-arch images (https://github.com/uselagoon/lagoon-images/issues/88) - the difficulty is with keeping local and production the same - what works for ARM may not work for AMD64 etc - so you'd have to be careful of any divergence.

Head over to the https://github.com/uselagoon/lagoon-images repo and see what we've got there - the makefile contains the build instructions (make clean && make -j6 build) - happy for any insights

ocean commented 3 years ago

Thanks @tobybellwood, thanks very much for the link to uselagoon image repo! 🎉 Can't believe I didn't find that one from the Hub org name 😁 I'll dive into it now.

I do agree the divergence between architecture versions may become an issue over time, though hopefully there's a way to handle that - I'm thinking conditionals etc, and I think Alpine is at least a great "bottom layer".

tobybellwood commented 3 years ago

Yeah - we've already had to look at a couple of dockerfile changes to support different upstreams for a couple of images that will have to be Debian-based going forwards - https://github.com/uselagoon/lagoon-images/blob/testing/upstream_solr_7_8/images/solr/7.Dockerfile#L31-L38

ocean commented 3 years ago

Interesting. I just discovered MongoDB isn't available on Alpine Linux past version 3.9, so I assume you'd be looking at moving that image to Debian as well maybe?

tobybellwood commented 3 years ago

MongoDB isn't available past 4.02 in Alpine 3.9 because of SSPL issues more likely. The only upstream MongoDB supported images are Ubuntu. We're currently looking at this (and some others) - noting that the infrastructure MongoDB providers only commonly support 3.6 still...

ocean commented 3 years ago

Yep, it is unavailable in Alpine because of licensing issues.

ocean commented 3 years ago

@tobybellwood So, it was a heck of an adventure, but I got a functioning, minimalist stack of Lagoon + GovCMS containers running on my Apple M1 MBP! 🎉 🎉

I'll put in some PRs where I can, and am really wanting to contribute as much as possible back, but in summary quite a lot of it was rebuilding some of the many small tools (mostly Go ones) to run on linux/arm64, and then using Docker Moby Buildkit to build linux/arm64 lower level images to base my containers off.

I've put a PR in for cross-compiling ahoy, and will put in more for docker-gen, envplate, dockerize, trivy, pygmy and the docker-haproxy image once I get proper multi-arch builds going for them.

tallytarik commented 3 years ago

@ocean curious how you've gone getting the set of containers running on M1?

I've been testing it out and running into errors with containers trying to start - redis/mariadb/nginx/php. Looks like there's an issue with Go, which means envplate doesn't run, which means these containers have invalid config.

Did you run into this by any chance?

ocean commented 3 years ago

@tallytarik Yep, that's correct, envplate was the first thing that died for me. It's because the envplate utility is in the commons Docker image, which is used as a utility parent image for most of the other ones. The ep Go binary in the container is compiled for amd64, and even though Docker for Mac is emulating amd64, it must not support the calls the binary is making. Luckily, Go binaries are really easy to recompile on various architectures, so I did this as part of the Dockerfile build.

I created my own fork of this repo where I updated the Dockerfiles to work cross platform. I haven't made a PR in this repo yet because there didn't previously seem to be a lot of interest, but perhaps now that Docker for Mac on M1 is in full release there will be more people looking to run Lagoon on it.

Here's my commons Dockerfile, in my fork which has most of the Lagoon stack in it working now (essentially I stuck to the images required to run GovCMS, as that's what I needed most urgently):

https://github.com/ocean/lagoon-images/blob/main/images/commons/Dockerfile

https://github.com/ocean/lagoon-images/

All my Docker images from this repo have been built for M1 ( linux/arm64 ) and are in my Docker repo here: https://hub.docker.com/u/oceaniclagoon

And the GovCMS ones I built are here: https://hub.docker.com/u/oceanicgovcms (source for these is at https://github.com/ocean/lagoon )

ocean commented 3 years ago

@tallytarik Oh I meant to say, the Lagoon pygmy stack has issues running in amd64 emulation mode too, so I rebuilt some of the Docker images that power that and then forked the original Ruby gem to use them.

Source is here https://github.com/ocean/pygmy

The Gem itself is called pygmy-apple and is available from RubyGems https://rubygems.org/gems/pygmy-apple/

  1. gem install pygmy-apple
  2. pygmy-apple up
ocean commented 3 years ago

Added a small PR to the lagoon-images repo to initiate discussion https://github.com/uselagoon/lagoon-images/pull/150

Feng-Shui commented 3 years ago

Two questions:

1: What type of beer/spirits/wine/coffee/tea do you drink? 2: How can I get you a carton of said drink?

Am up and running on these images this morning, you, sir, are a saint.

tobybellwood commented 3 years ago

I guess I know what I'm doing for the next few days then...

IMG_20210507_093027.jpg

ocean commented 3 years ago

@Feng-Shui No worries, I'm glad you got it all working 👍 Next time we cross paths (I plan to be in the East later in the year), I would be very happy to let you buy me a beer 😁

@tobybellwood I hope your weekend ARM hacking is successful 🤓

I realise now I should have called my fork pygmy-arm instead of pygmy-apple, my bad 😐 (and why the heck is it called pygmy anyway??)

Feng-Shui commented 3 years ago

I wondered the same thing for ages, name is explained in the docs: https://pygmy.readthedocs.io

tobybellwood commented 3 years ago

ok - a quick update, as this is getting close now (I did a heap of work on this over the last month, but have been dragged away again):

Next steps - if anyone is game to play with their local envs to help test this (esp on M1) - let me know, and you can build my version of pygmy-go (as pygmy-go-dev alongside your normal install), change your FROMs in your dockerfile and let me know how it goes...

ajaxmac commented 3 years ago

@tobybellwood Thanks for the update. I am keen to test this out, as I am blocked after purchasing a new M1 iMac, and I cannot run my lagoon based projects.

I pulled the dev branch of the docker_haproxy repo you listed above. And ran docker compose up and it all went well until I got a 

Error response from daemon: driver failed programming external connectivity on endpoint docker-haproxy_haproxy_1 (3ec39e176ff22a5157b43201d4c97672e39d3408233425a72c276049fe697438): Bind for 0.0.0.0:443 failed: port is already allocated

When I run 

lsof -i tcp:443 | grep LISTEN

There's nothing listening on port 443.

If I get this up and running. How do I integrate with my existing projects?

tobybellwood commented 3 years ago

I'm going to assume that there's another pygmy running somewhere using 443 (docker ps | grep 443) - it doesn't show in lsof :scream:

The best way to test it is to get my pygmy-go fork built - that will pull the correct images in from upstream.

If you're already running pygmy (or pygmy-go) make sure to stop them first

These steps should get it up and running - make sure to use the correct binary darwin-arm64 for M1

pygmy down && pygmy-go down
git clone --branch arm_testing git@github.com:tobybellwood/pygmy-go.git ./pygmy-go-dev
cd ./pygmy-go-dev
make build
cd builds
./pygmy-go-darwin-arm64 status
./pygmy-go-darwin-arm64 up

And that should bring up a fresh pygmy-go stack, built on the new images.

(note on Ubuntu/Linux you may get a bind: permission denied error) - to fix this, run sudo setcap 'cap_net_bind_service=+ep' /home/tobybellwood/sites/pygmy-go/builds/pygmy-go-linux-amd64-static

Feng-Shui commented 3 years ago

I've got the stack running locally but I can't get pygmy to add my ssh key, I've posted a comment on a similar issue in Karl's original repo and tagged you in it:

https://github.com/fubarhouse/pygmy-go/issues/312#issuecomment-931025435

Feng-Shui commented 3 years ago

pygmy-apple can add my key but interestingly, I seem to have lost ssh access to Lagoon instances. "no access to [project]-[env]".

I was seeing some ssh errors early this morning when trying to push/pull/fetch from govCMS GitLab (approx 5:30am).

Connection closed by 3.105.0.127 port 22
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

That's not an issue tonight, but I wonder if there's some other ssh weirdness going on today? Will check again in the morning.

ocean commented 3 years ago

@Feng-Shui Has your account expired in GovCMS GitLab?

Also, I'd say move to @tobybellwood 's pygmy-go installation on your M1 now, it's more up to date than mine:

https://github.com/tobybellwood/pygmy-go/tree/arm_testing

Feng-Shui commented 3 years ago

Nah, it was an issue with the SSH pods, was resolved this morning.

Toby's fork is what I tried to get running last night, it all worked except it wouldn't add my ssh key, although based on that issue in Karl's repo, I'm not sure if it's the fork or not.

I'll have a bit more of a poke around this weekend. At this point, I'm running on your pygmy-apple and Toby's multiarch Lagoon images.

tobybellwood commented 3 years ago

Thanks both - we're soooo close to releasing - just ironing a couple of last problems out!

ocean commented 3 years ago

@tobybellwood Great to hear! Anything I can take a look at that you'd like a random extra viewpoint on?

Feng-Shui commented 3 years ago

Morning, have done some testing on the new images and I'm seeing some performance issues with the mariadb image.

Running a db-write-heavy set of drush commands on the testlagoon images (testlagoon/mariadb-drupal: arm64-images) I'm getting a execution time of 951 seconds, switching out mariadb for oceanicgovcms/mariadb-drupal (and keeping the rest of the testlagoon images) this drops to 451 seconds (which is normal).

@tobybellwood you seen this, and/or should I be used a different tag?

tobybellwood commented 3 years ago

@Feng-Shui how about with the testlagoon/mariadb-10.5-drupal:multiarch ?

Feng-Shui commented 3 years ago

Will test now and let you know.

Feng-Shui commented 3 years ago

469 seconds with testlagoon/mariadb-10.5-drupal:multiarch 🚀

tobybellwood commented 3 years ago

the mariadb-drupal image without the version number is still pointing at 10.4, but we're going to try to move people onto versioned images (because they're much safer) - so that one in particular isn't slated to be made multi-arch, so you were seeing the amd64 image via the emulation.

tobybellwood commented 3 years ago

We've got a lot of documentation to do about this change, because we only want people to have to change tags once, and hopefully make everyone's lives easier!

tobybellwood commented 2 years ago

Lagoon Images are now fully ARM-compatible