search

rake-compiler / rake-compiler-dock

Easy to use and reliable cross compiler environment for building Windows, Linux, Mac and JRuby binary gems.
MIT License
77 stars 30 forks source link

Use manylinux Docker image for aarch64-linux #68

Closed stanhu closed 2 years ago

stanhu commented 2 years ago

Previously Ubuntu 20.04 was used to build aarch64-linux, which caused native gems to require glibc v2.29 or higher. However, this prevented native gems from being used on Amazon Linux 2, Debian 10, and more.

To improve this, let's use the manylinux 2014 image as we do with the x86 builds. Since this image is based off CentOS 7, which uses glibc 2.16, this significantly improves the compatiblity of ARM64 builds.

Relates to https://github.com/sparklemotion/nokogiri/issues/2470

flavorjones commented 2 years ago

I've kicked off CI.

Looking at the commit history, I think that I erred by not using manylinux when adding aarch64-linux in c5358ea (it was already in use at that time for all the other linux builds (since 8e85ee8)).

And now I wonder if we should use manylinux for arm-linux as well ... let's tackle that in a separate PR.

flavorjones commented 2 years ago

So we can see the challenge in the failed build, @stanhu -- the manylinux aarch64 docker image contains aarch64 binaries, which won't run on x86 hosts (like github actions and the majority of linux dev machines as of 2022-04). All the other docker images will run on x86 hosts.

I have this same problem running on my dev machine. I'm curious if you tested this locally? Were you able to get it to work without using qemu?

I think it might be possible to package multiarch/qemu-user-static into this Docker container, if you're up for it.

stanhu commented 2 years ago

@flavorjones Yeah, I wasn't able to build on my x86 Linux machine, but I think this image built fine on my Apple M1 machine?

I've attempted to build under QEMU for this platform in 53d541c. Is there any way I can kick off CI for my own testing here? UPDATE: Oh, I see you have to approve. 😄

flavorjones commented 2 years ago

I kicked off CI again (because you're a first-time contributor, the repo's CI setting requires me to push a button).

flavorjones commented 2 years ago

@stanhu It looks like the docker image builds OK with the current changes ... but it feels like the changeset may be larger than necessary. On my dev machine, simply running docker run --rm --privileged multiarch/qemu-user-static --reset -p yes --credential yes allows me to build the image (run everything in the Dockerfile).

Is that something you tried on Github Actions in one of the iterations already? It's a much simpler change and so I assume I'm missing something, but wanted to ask.

flavorjones commented 2 years ago

I want to gut-check with @larskanis about how he feels about needing to run QEMU (or equivalent) in order to build gems in the aarch64-linux container. I personally am a little concerned with how slow gem building is (I'm running a Nokogiri build on my dev machine with the manylinux container) and the additional configuration of QEMU may be challenging for some users.

stanhu commented 2 years ago

I want to gut-check with @larskanis about how he feels about needing to run QEMU (or equivalent) in order to build gems in the aarch64-linux container. I personally am a little concerned with how slow gem building is (I'm running a Nokogiri build on my dev machine with the manylinux container) and the additional configuration of QEMU may be challenging for some users.

Yeah, the Docker image under QEMU is taking over an hour to build. This is too long. How does the Ubuntu 20.04 build work here? Is it cross-compiling for aarch64-linux?

larskanis commented 2 years ago

To be honest, I don't like the approach. Manylinux has a different philosophy: they only target linux and don't cross build, while RCD is all about cross build, for different target platforms, but all running on x86_64-linux. Manylinux also has the downside, that it uses a different distribution (Centos) than the other target platforms (Ubuntu) and so gem projects have to use different commands to do the same things (apt vs. yum) depending on the platform. Manylinux also increases the size of the images a lot and docker layer caching/reusing can not be used. Now we have another downside, in such a way, that it's awfully slow.

So rather than increasing the use of Manylinux I'd like to remove it and instead use an approach similar to the x64-mingw-ucrt image: We build our own cross build toolchain in a separate docker image and copy the necessary files into our distributed image for x86_64 / aarch64 linux platforms.

stanhu commented 2 years ago

Thanks, I closed this since it's way too slow.

So rather than increasing the use of Manylinux I'd like to remove it and instead use an approach similar to the x64-mingw-ucrt image: We build our own cross build toolchain in a separate docker image and copy the necessary files into our distributed image for x86_64 / aarch64 linux platforms.

Is x64-mingw-ucrt is based off Ubuntu 20.04? How does this approach ensure that the binary is compiled with older glibc versions? Is there a reason Ubuntu 18.04 can't be the default instead of 20.04?

stanhu commented 2 years ago

@larskanis We got bitten by the glibc v2.29 issue again in https://github.com/sparklemotion/nokogiri/issues/2470. Could you look at my questions above ~and point me at the x64-mingw-ucrt Dockerfile~?

UPDATE: I see it.