search

mstorsjo / llvm-mingw

An LLVM/Clang/LLD based mingw-w64 toolchain
Other
1.75k stars 176 forks source link

wip: build linux/arm64 docker images #399

Closed dtretyakov closed 4 months ago

dtretyakov commented 5 months ago

Fixes #398 by making docker image multi-arch with linux/amd64 and linux/arm64 platforms.

mstorsjo commented 5 months ago

Unfortunately, this solution doesn't work in practice on github actions.

The proposed solution works by running the whole docker build for the foreign architectures within QEMU. For small projects, this might be acceptable, but for something on the size of LLVM, it makes things way way too slow. The last release build where I ran these jobs, the jobs took almost 3 hours to build, when building entirely natively - see e.g. https://github.com/mstorsjo/llvm-mingw/actions/runs/7712982313. The timeout for github actions jobs is 6 hours. I have no idea how long this would take to build in QEMU.

I tested out your PR in a branch, where I modified the github actions job to run on regular push (instead of triggered manually, e.g. on release): https://github.com/mstorsjo/llvm-mingw/commits/docker-arm64 As expected, it timed out: https://github.com/mstorsjo/llvm-mingw/actions/runs/7766688347/job/21182780468 Here, it had progressed to build step 784/5833 when it timed out. If we would have skipped the regular native build and just do the cross build for arm64 in a job of its own, it would certainly get further, but I'm pretty sure it would still take way too long.

I think a more practical way to solve this, would be to fetch the artifacts from the regular build, with dawidd6/action-download-artifact like we already do in the prepare step, but instead fetch the linux-ucrt-aarch64-toolchain artifact. Then we'd use a separate Dockerfile, which just unpacks this existing tarball in the right place. That should be quite manageable.

dtretyakov commented 5 months ago

@mstorsjo I applied the changes where we're downloading toolchains as build artifacts and using them to produce multi-arch images.

mstorsjo commented 4 months ago

@mstorsjo I applied the changes where we're downloading toolchains as build artifacts and using them to produce multi-arch images.

Thanks! This looks pretty good, and almost works. It needs one small tweak; when downloading more than one artifact, both with actions/download-artifact and dawidd6/action-download-artifact, they are extracted into a subdirectory - see https://github.com/mstorsjo/llvm-mingw/commit/75ab9c5fb4a298860b51968ec6afb2381d49d138.

Additionally, I tested a few other modifications. While we can create the regular x86_64 docker image this way, I'd prefer to keep producing it with the full dockerfile like we used to. If you don't mind, I would prefer to update this PR with those changes; see the changes in https://github.com/mstorsjo/llvm-mingw/compare/master...9a402d8af586fe184aaee0183f2d83fc3f04127f.

With that added/tweaked, I'd be happy to merge this PR!

mstorsjo commented 4 months ago

Update: I did a couple more tweaks, to try to run the toolchain once after unpacking, to guard against having a too old version of glibc/libstdc++, making the docker image unusable - new commit range: https://github.com/mstorsjo/llvm-mingw/compare/master...a17bfeb42b58a0f1669ed06bcb618f0423bc0061

dtretyakov commented 4 months ago

@mstorsjo thanks for moving it forward. You could consider this PR as a prototype and close it once you will get a working build in the arm64 branch.

mstorsjo commented 4 months ago

@mstorsjo thanks for moving it forward. You could consider this PR as a prototype and close it once you will get a working build in the arm64 branch.

Ok, great! I've tested most aspects of it, except for actually pushing the built image. I've updated the PR with my changes, and I'll go ahead and merge it. By the next release (LLVM 18.1.0 RC 3) in 2 weeks, such an image should get built and pushed.

mstorsjo commented 4 months ago

It turns out that we can’t do multi-platform images like this - when uploading one image for one architecture, it overwrites the old one with that same name. So we need a different upload procedure, including merging of the images - see https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners.