search

devcontainers / cli

A reference implementation for the specification that can create and configure a dev container from a devcontainer.json.
https://containers.dev
MIT License
1.52k stars 214 forks source link

ERROR: oci for multi-node builds currently not supported #524

Open DavidS-ovm opened 1 year ago

DavidS-ovm commented 1 year ago

When using a buildx builder for multi-arch building that uses actual native hosts (e.g. through kubernetes) to efficiently build on native CPUs, devcontainer build fails with

ERROR: oci for multi-node builds currently not supported

full logs

Run devcontainers/ci@v0.3
Starting...
Installing @devcontainers/cli...
** Installing @devcontainers/cli
/usr/bin/bash -c npm install -g @devcontainers/cli@0

added 3 packages in 6s
Adding --cache-from ghcr.io/overmindtech/devcontainer:latest to build args
🏗️ build container
  About to run devcontainer build --workspace-folder /home/runner/work/workspace/workspace/.devcontainer/prebuild --image-name ghcr.io/overmindtech/devcontainer:latest --platform linux/amd64,linux/arm64 --output type=oci,dest=/tmp/output.tar --cache-from ghcr.io/overmindtech/devcontainer:latest --cache-from ghcr.io/overmindtech/devcontainer
  [2023-05-17T11:09:59.174Z] @devcontainers/cli 0.41.0. Node.js v18.16.0. linux 5.15.0-1037-azure x64.

  [2023-05-17T11:10:02.840Z] Start: Run: docker buildx build --platform linux/amd64,linux/arm64 --output type=oci,dest=/tmp/output.tar --build-context dev_containers_feature_content_source=/tmp/devcontainercli-runner/container-features/0.41.0-1684321800429 --build-arg _DEV_CONTAINERS_BASE_IMAGE=mcr.microsoft.com/devcontainers/go:1.20 --build-arg _DEV_CONTAINERS_IMAGE_USER=root --build-arg _DEV_CONTAINERS_FEATURE_CONTENT_SOURCE=dev_container_feature_content_temp --target dev_containers_target_stage -t vsc-prebuild-47ba38bb9c7dba2aad678f24932563874c19808828a33025da1cde65e5475ab0-features -f /tmp/devcontainercli-runner/container-features/0.41.0-1684321800429/Dockerfile.extended /tmp/devcontainercli-runner/empty-folder

  [2023-05-17T11:10:06.020Z] ERROR: oci for multi-node builds currently not supported

  [2023-05-17T11:10:06.025Z] Exit code 1

Workaround

use qemu and patience instead of kubernetes backend

Following the instruction in https://dev.to/aws-builders/using-docker-manifest-to-create-multi-arch-images-on-aws-graviton-processors-1320 I was also able to build the x86 and arm64 images in two separate jobs and then assemble it afterwards in a separate step. Note that docker 20.10 (the default on github runners) is too old for docker manifest and I needed to upgrade to a newer version (24.0.0 and 23.0.6 both worked, see https://stackoverflow.com/a/59797984/4918) to get it to work.

timnolte commented 10 months ago

@DavidS-ovm do you have a documented example of how you took the results of the builds and combined them? I'm thinking that my problem right now is that I'm using the devcontainers/ci GitHub Action on dedicated native platform runners but the underlying devcontainer build command is using an --output type=oci,dest=/tmp/output.tar flag setup that I think is preventing me from properly using buildx to created the combined platform manifest/image.

I was of course able to just use QEMU, and like you pointed out wait a long time for builds, but the result is 45+ minute builds instead of like 15 minute builds(my arm64 runners are low on resources).

DavidS-ovm commented 10 months ago

@timnolte I was using the docker manifest command as documented in the link in the OP. Eventually I moved on to use https://depot.dev instead. It's a much nicer experience!