Open antonysouthworth-halter opened 11 months ago
Actually, an extra question expanding on the above, it seems like there is no way to provide credentials to the buildkit daemon at all? E.g. Docker Hub or ECR or other public or private registry.
For some additional context; some of our CI builds were failing due to exhausting the available free rate limit for Docker Hub, so we thought we could just cache the base image we are using in ECR and then update our Dockerfiles to use it, or e.g. set up Pull-Through cache for ECR, but ran into the above issue.
We were using the python:3.9-slim-buster
image which is available in public.ecr.aws so we just switched to that as a workaround.
We had a similar problem. We needed to pull an image from an AWS private repository through the FROM
instruction in a Dockerfile.
The workaround we ended up using was installing the aws-cli and docker, logging in, and then building the image. Below is an example:
Pipeline:
(...)
- task: build
privileged: true
config:
platform: linux
image_resource:
type: registry-image
source:
repository: concourse/oci-build-task
params:
DOCKERFILE: project/Dockerfile
CONTEXT: project/
AWS_ACCESS_KEY_ID: ((data.aws_key))
AWS_SECRET_ACCESS_KEY: ((data.aws_secret))
AWS_DEFAULT_REGION: ((data.aws_region))
inputs:
- name: project
run:
path: project/build.sh
project/build.sh:
#!/usr/bin/env sh
apk add aws-cli docker
docker login --username AWS --password "$(aws ecr get-login-password --region $AWS_DEFAULT_REGION)" <repository endpoint>
build
Being able to pull an image from a private repository seems like a great feature to add, since it's a very common use case, just like concourse/registry-image-resource.
@antonysouthworth-halter @pedrodsrodrigues You can use a docker build argument ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image
and pass it to the oci-build-task params with IMAGE_ARG_BASE_IMAGE and=docker-base/image.tar
The Dockerfile should look like so
ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image
ARG BASE_IMAGE_TAG=my-tag
FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG}
....
And on the CI pipeline
---
resource:
# Docker images
- name: docker-base
type: registry-image
source:
repository: apptweak-app
tag: base
aws_access_key_id: ((AWS_ACCESS_KEY_ID))
aws_secret_access_key: ((AWS_SECRET_ACCESS_KEY))
aws_region: ap-southeast-2
...
jobs:
- name: build-app
plan:
- in_parallel:
- get: git-app
params:
depth: 1
trigger: true
- get: docker-base
params:
format: oci
- task: build-image
privileged: true
config:
platform: linux
image_resource:
type: registry-image
source:
repository: concourse/oci-build-task
username: ((DOCKERHUB_USERNAME))
password: ((DOCKERHUB_PASSWORD))
params:
IMAGE_ARG_BASE_IMAGE: docker-base/image.tar
inputs:
- name: git-app
path: .
- name: docker-base
outputs:
- name: built-image
run:
path: build
But I aggree, being able to pull the private image directy from the oci-build-task run will be more convenient. 😄
@antonysouthworth-halter @pedrodsrodrigues You can use a docker build argument
ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image
and pass it to the oci-build-task params withIMAGE_ARG_BASE_IMAGE and=docker-base/image.tar
The Dockerfile should look like so
ARG BASE_IMAGE=xxxxxxxxxxxx.dkr.ecr.ap-southeast-2.amazonaws.com/my-private-image ARG BASE_IMAGE_TAG=my-tag FROM ${BASE_IMAGE}:${BASE_IMAGE_TAG} ....
And on the CI pipeline
--- resource: # Docker images - name: docker-base type: registry-image source: repository: apptweak-app tag: base aws_access_key_id: ((AWS_ACCESS_KEY_ID)) aws_secret_access_key: ((AWS_SECRET_ACCESS_KEY)) aws_region: ap-southeast-2 ... jobs: - name: build-app plan: - in_parallel: - get: git-app params: depth: 1 trigger: true - get: docker-base params: format: oci - task: build-image privileged: true config: platform: linux image_resource: type: registry-image source: repository: concourse/oci-build-task username: ((DOCKERHUB_USERNAME)) password: ((DOCKERHUB_PASSWORD)) params: IMAGE_ARG_BASE_IMAGE: docker-base/image.tar inputs: - name: git-app path: . - name: docker-base outputs: - name: built-image run: path: build
Great solution, thanks for sharing! Unfortunately we are using multi-arch images, which seems kind of incompatible with IMAGE_ARG
: https://github.com/concourse/oci-build-task/issues/110
I was able to have the oci-build-task successfully pull from a private registry. The workaround I tried was two fold.
.docker/config.json
file containing the auth required.DOCKER_CONFIG
in the oci-build-task to dictate which directory should be used to look for this auth file(https://docs.docker.com/engine/reference/commandline/cli/#configuration-files)jobs:
- name: hello-world
plan:
- task: login
config:
platform: linux
image_resource:
type: registry-image
source:
repository: alpine
outputs:
- name: docker-config
run:
path: /bin/sh
args:
- -c
- |-
mkdir -p docker-config
cat <<EOF > docker-config/config.json
{
"auths": {
"<some-registry-server>": {
"auth": "<some-base64-encoded-string>"
}
}
}
EOF
- task: oci-build
privileged: true
config:
inputs:
- name: docker-config
platform: linux
image_resource:
type: registry-image
source:
repository: concourse/oci-build-task
params:
OUTPUT_OCI: true
DOCKER_CONFIG: docker-config
run:
path: build
Describe the bug
I have a Dockerfile that looks like this:
The build falls over pretty much immediately:
As you can see, I already ensure the credentials are passed to
build
so I dunno why I still get 401.Reproduction steps
Run build with the above Dockerfile.
Expected behavior
It should not fail on FROM statement.
Additional context
No response