Docker buildx component missing in `standard:7.0`

[SunsetYe66]

Describe the bug A clear and concise description of what the bug is.

Refers to Docker release note:

From Docker Engine version 23.0.0, Buildx is distributed in a separate package: docker-buildx-plugin. In earlier versions, Buildx was included in the docker-ce-cli package. When you upgrade to this version of Docker Engine, make sure you update all packages.

This will result in Dockerfile stated BUILDKIT=1 to fail.

To Reproduce Steps to reproduce the behavior:

1. Add a new build project and set the image to standard:7.0
2. In buildspec.yml, use these lines:

- export DOCKER_BUILDKIT=1
- docker build .

4. Trigger build
5. See error:

   ERROR: BuildKit is enabled but the buildx component is missing or broken.
      Install the buildx component to build images with BuildKit:
      https://docs.docker.com/go/buildx/

Expected behavior This will success in both standard:6.0 and 5.0

Logs N/A

Platform (please complete the following information): N/A

Additional context N/A

[SunsetYe66]

@Dylan-AWS Can you check this? Docker buildx is essential for us since we need to mount secrets inside the container for specific steps, blocking us from migrating to standard:7.0. #626 is a possible fix for this problem and passed my local test, I hope that is helpful.

[Dylan-AWS]

We will look to include this on the next image release expected to be middle of May.

[Niknu]

@SunsetYe66 I did this to be able to add buildx into the standard 7.0

  # Note: The next 4 following step is done to be able to do step 5.
  # The solution is found here https://askubuntu.com/questions/1409192/cannot-install-docker-desktop-on-ubuntu-22-04
  - apt-get update && apt-get install ca-certificates curl gnupg lsb-release -y
  - mkdir -p /etc/apt/keyrings
  - curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
  - echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
  # NOTE: adding buildx to the the build command for docker: source  https://docs.docker.com/engine/release-notes/23.0/
  - apt-get update && apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin -y

[SunsetYe66]

@Niknu Appreciated. But I and my team think this plugin should be added on AWS side, not by manually install.

BTW, standard images straightforwardly download and decompress Docker files and does not used APT. There might be some possibility with your way to temporarily workaround this issue.

[Niknu]

@Niknu Appreciated. But I and my team think this plugin should be added on AWS side, not by manually install.

@SunsetYe66 , Totally agree with you. What I also mentioned before was also until this was fixed 😁

[markbaird]

It's also missing in aws/codebuild/amazonlinux2-aarch64-standard:3.0

By default I get this deprecation warning when running docker commands:

DEPRECATED: The legacy builder is deprecated and will be removed in a future release.
            Install the buildx component to build images with BuildKit:
            https://docs.docker.com/go/buildx/

If I set DOCKER_BUILDKIT=1 then I get:

ERROR: BuildKit is enabled but the buildx component is missing or broken.
       Install the buildx component to build images with BuildKit:
       https://docs.docker.com/go/buildx/           

[SunsetYe66]

@Dylan-AWS cc @subinataws

I think the fix is broken. Docker buildx binary also need to be located under /usr/local/lib/docker/cli-plugins

I tested in our account and with latest standard:7.0 version:

[Container] 2023/05/23 03:48:29 Running command export DOCKER_BUILDKIT=1
[Container] 2023/05/23 03:48:29 Running command docker build --build-arg BUILD_BASE_IMAGE=${BUILD_BASE_IMAGE}: 1.18-alpine --build-arg
RUN_BASE_IMAGE=${RUN_BASE_IMAGE}:3.9 --tag $REPOSITORY_URI:latest
ERROR: Buildkit is enabled but the build component is missing or broken.

Install the build component to build images with BuildKit:

https://docs.docker.com/go/buildx/

[Container] 2023/05/23 03:48:29 Command did not exit successfully docker build --build-arg BUILD_BASE_IMAGE=${BUILD_BASE_IMAGE}: 1.18-alpine --build-arg
RUN_BASE_IMAGE-${RUN_BASE_IMAGE}:3.9 --tag $REPOSITORY_URI:latest . exit status 1
[Container] 2023/05/23 03:48:29 Phase complete: BUILD State: FAILED
[Container] 2023/05/23 03:48:29 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: docker build --build-arg
BUILD_BASE_IMAGE=${BUILD_BASE_IMAGE}:1.18-alpine --build-arg RUN_BASE_IMAGE=${RUN_BASE_IMAGE}:3.9 --tag $REPOSITORY_URI:latest
Reason: exit status 1

[mikemucc]

Literally tried to use this earlier today and encountered exactly what is stated above.

Any update on the fix?

[Antonio-AWS]

I think the fix is broken. Docker buildx binary also need to be located under /usr/local/lib/docker/cli-plugins

This is accurate. The buildx plugin will be installed in the correct location with the next release.

In the meantime, buildx is currently installed in /usr/local/bin/docker-buildx for standard:7.0.

As a temporary workaround, you may be able to use buildx by either -

1. Calling buildx via docker-buildx (i.e docker-buildx ls) if calling buildx directly
2. Linking/moving the file to the correct directory and running the expected command

Example buildspec -

  - export DOCKER_BUILDKIT=1 #enable buildkit for using buildx.
  - ln -s /usr/local/bin/docker-buildx /usr/local/lib/docker/cli-plugins/docker-buildx
  - docker build ...

[jasonkaedingrhino]

Did this get fixed? We had to implement this symlink workaround for 7.0 image. However, that workaround failed today because "file already exists" which suggests that it has been fixed.

@Antonio-AWS can we get a confirmation?

[Antonio-AWS]

Fixed in latest release: https://github.com/aws/aws-codebuild-docker-images/commit/182bbfdc86ff9b35ddf5d96f9b9140c46af14438#diff-c1ea454e65b48efad7d52406144e50f8459244517f0eb25a5ec3da4f865a3129

[dil-anovosz]

Wondering why is this not included in amazonlinux2-aarch64?

[abhishek-parative]

@Antonio-AWS When will the amazonlinux2-aarch64 Image be fixed? https://github.com/aws/aws-codebuild-docker-images/issues/640