Doesn't Support Multi-Platform Builds w/ Platform Native Runners

[timnolte]

I'm trying to pre-build an image that supports both amd64 & arm64, additionally I'm generating a large matrix of versions(12 image variations will be built). My goal is to build the amd64 version of the images on native GitHub Runners and then the arm64 version of the images on a self-hosted arm64 GitHub Runner. There is some documentation about attempting this however I'm finding it a challenge to find the build results locally in the runners so that I can upload them for a later job that will generated the multi-platform manifest.

Here is the part of my workflow where I'm attempting to use skopeo to pull the container image to then upload as an artifact.

https://github.com/ndigitals/wp-dev-container/blob/fix/Container-Image-Copy/.github/workflows/release-build.yml#L79-L107

[timnolte]

So, it appears that the images aren't actually being created and/or kept if you don't push the images. This makes the GitHub Action completely unusable to properly use native platforms for creating multi-platform builds.

[timnolte]

After further review of the Dev Container build process it appears that the image is already being copied to a .tar archive.

devcontainer build --workspace-folder /home/runner/work/wp-dev-container/wp-dev-container --image-name ghcr.io/ndigitals/wp-dev-container:php-8.0-node-16-amd64 --platform linux/amd64 --output type=oci,dest=/tmp/output.tar --cache-from ghcr.io/ndigitals/wp-dev-container:php-8.0-node-16-amd64 --cache-from ghcr.io/ndigitals/wp-dev-container

I believe this may actually solve my issue where I should be able to just move & rename the archive to where I need it for artifact upload.

[timnolte]

So, this doesn't work. The output format created by the devcontainer CLI is oci and not the proper type for a digest to be used by buildx.

[trxcllnt]

• build each image in parallel
  • both using the same image name/tag
  • and passing --output type=docker,dest=${{ runner.temp }}/${{ inputs.arch }}.tar
• upload each image to whatever intermediate artifact storage solution you like
• then in a follow on job, download the tarballs from your intermediate artifact storage
• and finally, make the manifest yourself

That last step is a bit of dockerhub trickery that isn't guaranteed to work in all registries. It relies on the fact that dockerhub automatically untags (but doesn't delete) prior images when a new image is pushed to the same tag, so YMMV.

docker push echos the remote hash, which is what needs to go into the manifest. Then docker buildx imagetools create creates and pushes the manifest with the remote hashes.

An example:

# in amd64 runner
devcontainer build ... --image-name foo:123 && <upload to artifact storage>
# in arm64 runner
devcontainer build ... --image-name foo:123 && <upload to artifact storage>

# in follow-on job that pushes the images and manifest:

# load the amd64 variant of `foo:123`
docker load --input "amd64.tar"
# push the local image to the registry and save the remote hash
hashes+=("$(docker push foo:123 | tail -n1 | cut -d' ' -f3)");

# load the arm64 variant of `foo:123`
docker load --input "arm64.tar"
# push the local image to the registry and save the remote hash
# this overwrites the tag from pushing the amd64 variant
hashes+=("$(docker push foo:123 | tail -n1 | cut -d' ' -f3)");

# create and push the manifest
# this overwrites the tag from pushing the arm64 variant
docker buildx imagetools create --tag foo:123 ${hashes[@]};

[chrmarti]

@timnolte You can pass a comma-separated list of platforms to the action: platform: linux/amd64,linux/arm64. Note that if you happen to run it on Ubuntu 22.04, you will need to update the skopeo tool. (See, e.g., https://github.com/chrmarti/test-devcontainer-ci/blob/65c7ecb50e837dc64e17521448bf5f17ffc66c22/.github/workflows/test.yml#L70 which I have used for my testing.)

[timnolte]

@chrmarti the problem is being able to use native GitHub Action Runners for the architecture. Using that method requires QEMU and emulation which burns up tons of time, as in allotted GitHub Action Runner minutes, because the emulation is extremely slow.

I ended up having to setup separate steps, that target using a native runner using a matrix build. Then I had to manually use Docker buildx in a follow up job to create a new manifest that includes both image architectures into the same tags. The workflow that I ended up with is here:

https://github.com/ndigitals/wp-dev-container/blob/develop/.github/workflows/release-build.yml

It would sure be nice to pass to this GitHub Action the native runners and it would be able to do all of this automatically but it is a huge complicated setup in order to produce multiplatform images, as it also won't work just adding the comma separated list without also adding a QEMU setup for the required emulation to make it work.

[chrmarti]

@timnolte That is an interesting solution, thanks for sharing!

I'm not sure the action could spawn a runner for the other platform itself, spawning runners appears to be only possible as part of the workflow file's runs-on properties. Maybe your approach is indeed the only way to achieve this?