Buildkit and Harbor Proxy Cache

[kenotsolutions]

Due to hub docker rate limit policy

I had to install harbor 2.1 and arranged its a proxy cache. It is working as expected such as if I use it in my deployment.yaml files or directly pull from harbor cache there is no problem.

Docker pull has no problem. docker pull harbor-cache.abc.com/docker/library/centos:7

but when I try to build an image from Dockerfile I got below error.

FROM harbor-cache.abc.com/docker/library/centos:7  
RUN \
    yum install -y python36 python36-libs python36-devel python36-pip git  && \
    yum clean all
RUN pip3.6 install --upgrade pip

registry_url= "harbor.abc.com"
const buildkit_addr = "tcp://buildkitd.buildkit.svc.cluster.local:1234";
let buildctl_args = "--addr ${buildkit_addr} build --frontend=dockerfile.v0 --local context=/build/ --local dockerfile=/build/ "

buildctl ${buildctl_args} --output type=image,\\"name=${registry_url}\\",push=true`

rpc error: code = Unknown desc = failed to load cache key: harbor-cache.abc.com/docker/library/centos:7: not found

There is no error for the below dockerfile

FROM centos:7  
RUN \
    yum install -y python36 python36-libs python36-devel python36-pip git  && \
    yum clean all
RUN pip3.6 install --upgrade pip

Any suggestion?

Looks like a similar issue

[hansbogert]

I'm not sure why the above mentioned similar issue and https://github.com/moby/moby/issues/40262 are closed while they show a disparity in functionality between buildkit and the "normal" builder. I'm hitting this too, though with the normal docker build functionality with buildkit enabled.

[tonistiigi]

Buildkit v0.6 (vendored in 19.03) / Containerd 1.2 do not support mirrors that don't actually contain the data and return error (instead of forward to upstream in that case). Newer version support it. This is a possible explanation. I have no visibility what Harbor Proxy Cache actually does.

[hansbogert]

@tonistiigi

I hope the underlying issue is the same as with the OP, but with version 19.03.13, containerd 1.3.7; I still see this behaviour:

$ cat Dockerfile.test
FROM debian:10

$ DOCKER_BUILDKIT=1 docker build -f Dockerfile.test . 
[+] Building 1.0s (4/4) FINISHED                                                                                                                                                                                
 => [internal] load .dockerignore                                                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                                                            0.0s
 => [internal] load build definition from Dockerfile.test                                                                                                                                                  0.0s
 => => transferring dockerfile: 42B                                                                                                                                                                        0.0s
 => ERROR [internal] load metadata for docker.io/library/debian:10                                                                                                                                         0.6s
 => ERROR [1/1] FROM docker.io/library/debian:10                                                                                                                                                           0.3s
 => => resolve docker.io/library/debian:10                                                                                                                                                                 0.3s
------
 > [internal] load metadata for docker.io/library/debian:10:
------
------
 > [1/1] FROM docker.io/library/debian:10:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: docker.io/library/debian:10 not found

$ sudo cat /etc/docker/daemon.json
{
    "default-address-pools":[{
        "base":"172.29.0.0/16",
        "size":24
    }],
        "ipv6": true,
        "fixed-cidr-v6": "2001:db8:1::/64",
        "registry-mirrors": ["https://mirror.gcr.io"]
}

If you think this is a different issue, I'll gladly open a new issue.

[tonistiigi]

It is the vendored containerd library version that matters so 19.03 is always affected. Try with 20.10-beta or newer buildkit with docker buildx.

[hansbogert]

@tonistiigi Thanks a lot; this works!

But it seems I hijacked the thread then, sorry.