search

toltec-dev / toolchain

Set of Docker images for cross-compiling binaries targeting the reMarkable tablet.
MIT License
34 stars 12 forks source link

Add linux/arm64 platforms to image builds #30

Open timower opened 11 months ago

timower commented 11 months ago

This should allow the toolchain to be easily used on apple silicon macs (or any other arm64 device).

Closes #28

timower commented 11 months ago

I'll try something like https://docs.docker.com/build/ci/github-actions/multi-platform/#distribute-build-across-multiple-runners, otherwise I guess this isn't feasible

Eeems commented 11 months ago

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

beelux commented 7 months ago

GitHub seems to be adding M1 runners as part of their "Standard GitHub-hosted runners for Public repositories", but they are not (yet) listed on their "actions/runner-images" repo. They have an open, recently (in hours) approved, but not yet merged PR, so I'm guessing it's coming very soon!

The upside is that this is native arm64, which would most likely run faster than emulated arm64 on amd64! Thus, this could possibly solve the timeout issue.

The macOS images don't seem to have docker pre-installed, but there are reports of users simply installing it through homebrew The building scripts seem like they'd run on macOS just fine, as they're just bash with a bunch of docker commands.

beelux commented 7 months ago

I've tried to bodge something together to show it'd work, but I'm stuck on a QEMU HV_UNSUPPORTED issue when starting colima ("container runtimes on macOS"). The previous reports were on macos-12, while this is running on macos-14, which is a beta for GitHub.

I've opened an issue on the colima repo, but I'm guessing it might be a bug on GitHub's side.

beelux commented 7 months ago

My optimism in regards to M1 runners was short-sighted: GitHub (and simply Apple) does not support (nested) virtualization on those runners, which results in the VM not being able to boot up. There's thus no way to really improve the earlier amd64 Linux runner solution('s run time).

I found out about this through the issue I opened on the colima repo, and the announcement PR on actions/runner-images, which pointed at the GitHub docs:

(...) nested-virtualization is not supported by arm64 runners

Tl;dr of this whole ordeal

Stick with amd64 Linux runners (for the foreseeable future). GH doesn't seem to be currently working on Linux arm64 runners (https://github.com/actions/runner-images/issues/5631).

jandubois commented 7 months ago

GitHub (and simply Apple) does not support (nested) virtualization

Note that while GitHub claims it to be a limitation of Apple's Virtualization Framework, it really is a limitation of the M1 chip architecture; nested virtualization is supported by Apple's Virtualization Framework on M2 and M3 chips, which are based on a newer ARM spec.

beelux commented 7 months ago

To avoid timeouts due to how long it takes, probably. It might also make sense to split out the images into their own jobs as well. I've done that for the images I have generating nightly: https://github.com/Eeems/images/blob/main/.github/workflows/build.yml

I've looked at the dependencies between the images, and I've realized that one could have 3 different stages, with each image having it's own separate job. Using different jobs also allows it to go over the 6 hour limit.

Stage 1 would be the initial one, Stage 2 starts once toolchain is done, Stage 3 starts once base is done. Each of the blocks are separate jobs.

multi stage building

One would have to look into the exporting of images, to transfer them between jobs, for builds that are not published. One could also separate amd64 and arm64 builds, but I'm unsure if this is doable when publishing to the image repository.

Eeems commented 7 months ago

I would recommend moving dotnet6 to stage 3 as well, make that dynamic, and have toolchain and base be static steps.

beelux commented 6 months ago

I looked more into multi-platform Docker images, as I had some time.

Separating the building of the images per platform is possible (exporting/importing single-platform images is no issue), but combining the images at the end is slightly hacky, as Docker does not support direct publishing of multi-platform "images" (manifest), but rather forces the individual images to be published already before creating the multi-platform manifest. (https://github.com/docker/cli/issues/3350)

To get around this, one can publish the single-platform images onto a local registry ran through GHA services. Once they are published, create the manifest, and push it locally. Finally, it's possible to copy the multi-platform "image" to the public registry. (Similarly to https://github.com/rapidsai/gha-tools/pull/51)