search

CircleCI-Public / docker-orb

Install/use various Docker-related tools on CircleCI
https://circleci.com/orbs/registry/orb/circleci/docker
MIT License
29 stars 62 forks source link

Integrate buildx #134

Open zigarn opened 2 years ago

zigarn commented 2 years ago

Any plan to allow users to build with buildx?

gavinclarkeuk commented 1 year ago

@brivu any updates on this?

gavinclarkeuk commented 1 year ago

@brivu any updates on this?

brivu commented 1 year ago

Hey @gavinclarkeuk and @zigarn,

I'm currently working on fixing issues for this orb at the moment but will circle back to add buildx capabilities.

We just want to be thoughtful in how we approach this i.e specifying different contexts, using buildx with Docker Layer Caching and more.

I'll do my best to have this done in the next couple of months.

Best, Brian

filip-aipl commented 1 year ago

hey @brivu! Any update on the buildx? Is there any chance to use maybe dev version with buildx integrated?

brivu commented 1 year ago

Hey Everyone,

We're wrapping up the quarter here in the next couple of weeks and there aren't any plans to have this released. I'm hoping we'll prioritize this next quarter and will bring your feedback back to the team as we plan.

The good news is we have a fix in place to get DLC working with buildx, which was blocking us from implementing this in the first place.

Thanks for your patience here!

Best, Brian

gavinclarkeuk commented 1 year ago

@brivu any updates? its been a couple of months

gavinclarkeuk commented 1 year ago

@brivu any updates?

EricRibeiro commented 9 months ago

Hey all, sorry about the delay.

Before we proceed, I want to understand your use case and why the current implementation doesn't work for you. This will help us design the orb in a way that makes sense for as many people as possible. Please answer the following:

Thanks in advance!

troyharvey commented 9 months ago

@EricRibeiro Here's the job we are using that we would like to replace with the orb. There are two key features we're missing:

jobs:
  docker-buildx-push:
  # A job that can build/push ARM and AMD Docker images.
  # TODO: Use the Docker orb once buildx multi-arch support is added.
  # @see https://github.com/CircleCI-Public/docker-orb/issues/134
  # @see https://stackoverflow.com/questions/73938529/docker-multi-platform-builds-extremely-slow-for-arm64-on-gitlab-ci
  # @see https://gist.github.com/jruiz94/f4b7bbb67e0ff7c71e9c582c7673af0c
    docker:
      - image: cimg/base:current
    parameters:
      arch:
        enum:
          - arm
          - amd
        type: enum
      platform:
        enum:
          - linux/arm64/v8
          - linux/amd64
        type: enum
      resource-class:
        enum:
          - large
          - arm.large
        type: enum
      tag:
        type: string
      poetry-version:
        type: string
        default: *poetry-version
    resource_class: << parameters.resource-class >>
    steps:
      - setup_remote_docker
      - checkout
      - dockerhub-login
      - run: |
          export DOCKER_CLI_EXPERIMENTAL=enabled
          docker buildx create --name multiarch --driver docker-container --use
          docker buildx build \
            --platform << parameters.platform >> \
            --secret id=POETRY_HTTP_BASIC_REDACTED_PASSWORD,env=redacted \
            --build-arg POETRY_VERSION=<< parameters.poetry-version >> \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            --push \
            --cache-to type=registry,ref=redacted/$CIRCLE_PROJECT_REPONAME:cache-<< parameters.arch >>,mode=max \
            --cache-from type=registry,ref=redacted/$CIRCLE_PROJECT_REPONAME:cache-<< parameters.arch >> \
            --tag redacted/$CIRCLE_PROJECT_REPONAME:<< parameters.tag >>-<< parameters.arch >> .
      - docker-buildx-push:
          # Build the candidate image using an ARM64 machine
          name: docker-buildx-push-arm
          platform: linux/arm64/v8
          arch: arm
          resource-class: arm.large
          tag: *tag

      - docker-buildx-push:
          # Build the candidate image using an AMD64 machine
          name: docker-buildx-push-amd
          platform: linux/amd64
          arch: amd
          resource-class: large
          tag: *tag
deiga commented 6 months ago

We use bake extract configuration to separate file from the CI config

afifurrohman-id commented 5 months ago

@EricRibeiro Here's the job we are using that we would like to replace with the orb. There are two key features we're missing:

  • buildx build --platform for building ARM and AMD
  • buildx build --secret for preventing secrets from leaking into the image from CI through environment variables
jobs:
  docker-buildx-push:
  # A job that can build/push ARM and AMD Docker images.
  # TODO: Use the Docker orb once buildx multi-arch support is added.
  # @see https://github.com/CircleCI-Public/docker-orb/issues/134
  # @see https://stackoverflow.com/questions/73938529/docker-multi-platform-builds-extremely-slow-for-arm64-on-gitlab-ci
  # @see https://gist.github.com/jruiz94/f4b7bbb67e0ff7c71e9c582c7673af0c
    docker:
      - image: cimg/base:current
    parameters:
      arch:
        enum:
          - arm
          - amd
        type: enum
      platform:
        enum:
          - linux/arm64/v8
          - linux/amd64
        type: enum
      resource-class:
        enum:
          - large
          - arm.large
        type: enum
      tag:
        type: string
      poetry-version:
        type: string
        default: *poetry-version
    resource_class: << parameters.resource-class >>
    steps:
      - setup_remote_docker
      - checkout
      - dockerhub-login
      - run: |
          export DOCKER_CLI_EXPERIMENTAL=enabled
          docker buildx create --name multiarch --driver docker-container --use
          docker buildx build \
            --platform << parameters.platform >> \
            --secret id=POETRY_HTTP_BASIC_REDACTED_PASSWORD,env=redacted \
            --build-arg POETRY_VERSION=<< parameters.poetry-version >> \
            --build-arg BUILDKIT_INLINE_CACHE=1 \
            --push \
            --cache-to type=registry,ref=redacted/$CIRCLE_PROJECT_REPONAME:cache-<< parameters.arch >>,mode=max \
            --cache-from type=registry,ref=redacted/$CIRCLE_PROJECT_REPONAME:cache-<< parameters.arch >> \
            --tag redacted/$CIRCLE_PROJECT_REPONAME:<< parameters.tag >>-<< parameters.arch >> .
      - docker-buildx-push:
          # Build the candidate image using an ARM64 machine
          name: docker-buildx-push-arm
          platform: linux/arm64/v8
          arch: arm
          resource-class: arm.large
          tag: *tag

      - docker-buildx-push:
          # Build the candidate image using an AMD64 machine
          name: docker-buildx-push-amd
          platform: linux/amd64
          arch: amd
          resource-class: large
          tag: *tag

We should up this issue.

Since GitHub actions doesn't have arm for Linux, even they have M1 runner release on 01/2024 but, M1 cannot use for nested virtualization therefor they cannot use for install docker. See: https://github.com/actions/runner-images/issues/9460

This is still a problem when want to build multi platform docker image. Using QEMU is totally super slow, and this is what Circle CI solved with arm executor.

So the point is, we can simplify this approach by using this ORBs.

I hope you can look this @brivu

EricRibeiro commented 5 months ago

@brivu and I are no longer with CircleCI. I suggest you contact support to escalate this to the proper folks.

marboledacci commented 1 week ago

I don't understand what is required here exactly. The build command uses buildx and it allows to pass any custom arg beyond the explicitly supported using the extra_build_args parameter. Is this parameter not suitable for your scenario for some reason?