search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.16k stars 1.16k forks source link

[Dockerfile] BuildKit does not respect Docker Registry Mirrors defined in daemon.json #606

Closed tangtony closed 4 years ago

tangtony commented 6 years ago

Hi,

I have been trying out the BuildKit feature built into Docker 18.06, but building with DOCKER_BUILDKIT=1 seems to cause my builds to fail since I am behind a firewall and need to access the docker hub registry through a local mirror.

/etc/docker/daemon.json:

{
  "registry-mirrors": ["https://my-mirror.company.com"],
  "experimental": true
}

Dockerfile:

FROM golang:alpine
RUN echo "test"

docker build -t test .

$ docker build .
[+] Building 0.6s (3/4)                                                                                                                          
 => local://context (.dockerignore)                                                                                                         0.3s
 => => transferring context: 2B                                                                                                             0.0s
 => local://dockerfile (Dockerfile)                                                                                                         0.2s
 => => transferring dockerfile: 96B                                                                                                         0.0s
 => ERROR docker-image://docker.io/library/golang:alpine                                                                                    0.1s
 => => resolve docker.io/library/golang:alpine                                                                                              0.1s
------
 > docker-image://docker.io/library/golang:alpine:
------
failed to do request: Head https://registry-1.docker.io/v2/library/golang/manifests/alpine: unexpected EOF
tonistiigi commented 6 years ago

cc @dmcgowan

adamsteinbok commented 6 years ago

I'm also seeing this issue. insecure-registries seems to be ignored as well.

tangtony commented 5 years ago

Is this supposed to be fixed in 18.09? Because I just updated to 18.09 and I'm getting the following error when I try to build now with a registry mirror:

#3 [internal] load metadata for docker.io/library/alpine:latest
#3       digest: sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
#3         name: "[internal] load metadata for docker.io/library/alpine:latest"
#3      started: 2018-11-11 20:47:20.993963345 +0000 UTC
#3    completed: 2018-11-11 20:47:20.994574285 +0000 UTC
#3     duration: 610.94µs
#3        error: "parse https://my-mirror.company.com%2F/v2/library/alpine/manifests/latest: invalid URL escape \"%2F\""

#4 [internal] load metadata for docker.io/library/golang:alpine
#4       digest: sha256:299327d28eff710219f2e24597cfa9b226e8b1b0dc90f9e2122573004cfe837f
#4         name: "[internal] load metadata for docker.io/library/golang:alpine"
#4      started: 2018-11-11 20:47:20.994231588 +0000 UTC
#4    completed: 2018-11-11 20:47:20.994427308 +0000 UTC
#4     duration: 195.72µs
#4        error: "parse https://my-mirror.company.com%2F/v2/library/golang/manifests/alpine: invalid URL escape \"%2F\""

Edit: The registry mirrors seem to work as expected when configured via the daemon.json. But when configuring the registry mirrors via docker daemon arguments, the error above occurs.

lbndev commented 5 years ago

I too have a problem with my private registry when buildkit is enabled in Docker 18.09.

This is my daemon.json :

{
  "registry-mirrors": ["https://repo.mycompany.com"],
  "insecure-registries" : ["repo.mycompany.com"],
  "features":{
    "buildkit": true
  }
}

When trying to build a Dockerfile starting with :

# syntax = docker/dockerfile:experimental
FROM maven:3.5.3-slim

I am getting this (I had done "docker pull docker.io/docker/dockerfile:experimental" first):

me@ic:~/test$ docker build .
[+] Building 0.1s (6/11)
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 38B                                                                                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                                                         0.0s
 => ERROR [internal] load metadata for docker.io/library/maven:3.5.3-slim                                                                                                                                                               0.0s
 => ERROR [1/6] FROM docker.io/library/maven:3.5.3-slim                                                                                                                                                                                 0.0s
 => => resolve docker.io/library/maven:3.5.3-slim                                                                                                                                                                                       0.0s
 => CACHED [internal] helper image for file operations                                                                                                                                                                                  0.0s
 => [internal] load build context                                                                                                                                                                                                       0.0s
 => => transferring context: 79B                                                                                                                                                                                                        0.0s
------
 > [internal] load metadata for docker.io/library/maven:3.5.3-slim:
------
------
 > [1/6] FROM docker.io/library/maven:3.5.3-slim:
------
failed to do request: Head https://repo.mycompany.com/v2/library/maven/manifests/3.5.3-slim: x509: certificate signed by unknown authority

If I set buildkit to false in daemon.json, everything works fine.

Platform : Ubuntu 18.04.1 LTS, docker-ce 18.09.0, build 4d60db4

tonistiigi commented 5 years ago

^^ @AntaresS

thomas-riccardi commented 5 years ago

This also means the GCP Container Registry's Docker Hub mirror cannot be used with buildkit: https://cloud.google.com/container-registry/docs/using-dockerhub-mirroring

Scenario

$ cat /etc/docker/daemon.json
{
    "features": {"buildkit": true},
    "debug": true,
    "registry-mirrors": ["https://mirror.gcr.io"]
}
$ cat Dockerfile 
FROM ubuntu:18.04
$ docker build .
[+] Building 0.5s (4/4) FINISHED                                                                                                                                                                                                             
 => [internal] load .dockerignore                                                                                                                                                                                                       0.0s
 => => transferring context: 2B                                                                                                                                                                                                         0.0s
 => [internal] load build definition from Dockerfile                                                                                                                                                                                    0.0s
 => => transferring dockerfile: 61B                                                                                                                                                                                                     0.0s
 => ERROR [internal] load metadata for docker.io/library/ubuntu:18.04                                                                                                                                                                   0.3s
 => ERROR [1/1] FROM docker.io/library/ubuntu:18.04                                                                                                                                                                                     0.2s
 => => resolve docker.io/library/ubuntu:18.04                                                                                                                                                                                           0.2s
------
 > [internal] load metadata for docker.io/library/ubuntu:18.04:
------
------
 > [1/1] FROM docker.io/library/ubuntu:18.04:
------
docker.io/library/ubuntu:18.04 not found

docker daemon logs:

janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.788539962+01:00" level=debug msg="Calling GET /_ping"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789372744+01:00" level=debug msg="Calling POST /session"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789389343+01:00" level=debug msg="Calling POST /v1.39/build?buildargs=%7B%7D&buildid=0339f145e333056de3753e4c03e661987b94d15db2be555233bffa2a367dd041&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=&labels=%7B%7D&memory=0&memswap=0&networkmode=default&remote=client-session&rm=1&session=aye173ehx136zr83ojvc807lh&shmsize=0&target=&ulimits=null&version=2"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789498322+01:00" level=info msg="parsed scheme: \"\"" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789512022+01:00" level=info msg="scheme \"\" not registered, fallback to default scheme" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789613181+01:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{ 0  <nil>}]" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789657431+01:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789726700+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, CONNECTING" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.789859839+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, READY" module=grpc
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.807385619+01:00" level=debug msg="reusing ref for local: sxtbvtno9sjb2fi3gsalzxu40"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.808593608+01:00" level=debug msg="diffcopy took: 1.05459ms"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.809481659+01:00" level=debug msg="reusing ref for local: y63o1okwyn5m255kfzv0ospr0"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.810173532+01:00" level=debug msg="diffcopy took: 605.114µs"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.813437291+01:00" level=debug msg="saved sxtbvtno9sjb2fi3gsalzxu40 as local.sharedKey:dockerfile:Dockerfile:8521e1980bcd1479d3a309f95d273cb76bca41fcbe4b5da4a8e8760f345728ee"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.819195975+01:00" level=debug msg="saved y63o1okwyn5m255kfzv0ospr0 as local.sharedKey:context:.dockerignore:8521e1980bcd1479d3a309f95d273cb76bca41fcbe4b5da4a8e8760f345728ee"
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.880379484+01:00" level=debug msg=resolving
janv. 14 11:27:10 myhostname dockerd[31946]: time="2019-01-14T11:27:10.880436003+01:00" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.141929136+01:00" level=debug msg="fetch response received" response.headers="map[Content-Type:[application/json] Server:[Docker Registry] Cache-Control:[private] X-Frame-Options:[SAMEORIGIN] Alt-Svc:[quic=\":443\"; ma=2592000; v=\"44,43,39,35\"] Accept-Ranges:[none] Docker-Distribution-Api-Version:[registry/2.0] Date:[Mon, 14 Jan 2019 10:27:11 GMT] X-Xss-Protection:[1; mode=block] Vary:[Accept-Encoding]]" status="404 Not Found" url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.142439491+01:00" level=debug msg=resolving
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.142480801+01:00" level=debug msg="do request" request.headers="map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]]" request.method=HEAD url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377198801+01:00" level=debug msg="fetch response received" response.headers="map[X-Frame-Options:[SAMEORIGIN] Vary:[Accept-Encoding] Docker-Distribution-Api-Version:[registry/2.0] Content-Type:[application/json] Date:[Mon, 14 Jan 2019 10:27:11 GMT] Server:[Docker Registry] Cache-Control:[private] X-Xss-Protection:[1; mode=block] Alt-Svc:[quic=\":443\"; ma=2592000; v=\"44,43,39,35\"] Accept-Ranges:[none]]" status="404 Not Found" url="https://mirror.gcr.io/v2/library/ubuntu/manifests/18.04"
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377730726+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, TRANSIENT_FAILURE" module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377763406+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, CONNECTING" module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377765596+01:00" level=warning msg="grpc: addrConn.createTransport failed to connect to { 0  <nil>}. Err :connection error: desc = \"transport: Error while dialing only one connection allowed\". Reconnecting..." module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.377807735+01:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc421cdb870, TRANSIENT_FAILURE" module=grpc
janv. 14 11:27:11 myhostname dockerd[31946]: time="2019-01-14T11:27:11.789751463+01:00" level=warning msg="grpc: addrConn.transportMonitor exits due to: context canceled" module=grpc

Actual behavior

The daemon seems to try to get image info 2 times from the same registry: the GCP mirror, which returns 404.

Expected behavior

The daemon should get image info from the GCP mirror first, and on 404 response try again on docker official registry.

Versions

$ cat /etc/os-release | grep VERSION
VERSION="18.04.1 LTS (Bionic Beaver)"

$ docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:49:01 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:16:44 2018
  OS/Arch:          linux/amd64
  Experimental:     false
tonistiigi commented 5 years ago

Tried with 18.09 and 19.03 with 

 cat /etc/docker/daemon.json
{
    "features": {"buildkit": true},
    "debug": true,
    "registry-mirrors": ["https://mirror.gcr.io"]
}

and had no issues. Verified from the logs that mirror was being used.

There are some workarounds in 19.03 to avoid codepath for https://issuetracker.google.com/issues/123043691 but as it works in 18.09.4 as well then I'm not sure if there was some gcr side change.

~ # docker version
Client: Docker Engine - Community
 Version:           18.09.4
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        d14af54
 Built:             Wed Mar 27 18:33:40 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.4
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       d14af54
  Built:            Wed Mar 27 18:41:10 2019
  OS/Arch:          linux/amd64
  Experimental:     false
SuperSandro2000 commented 5 years ago

I ran into this problem while testing the new buildkit UI on travis to measure how long each build step takes. Travis uses the https://mirror.gcr.io mirror and upgrading to 19.03 on testing solved my problem. Edit: No, it didn't. I cached the image while testing.

jamiejackson commented 4 years ago

I use a local, HTTP, pull-through registry. I seem to have the same problem after having enabled buildkit.

Docker 18

[vagrant@docker-host exp]$ cat /etc/docker/daemon.json
{
    "bip": "172.17.0.1/16",
    "debug": false,
    "dns": [
        "8.8.8.8",
        "10.80.1.10",
        "10.80.1.11"
    ],
    "experimental": true,
    "features": {
        "buildkit": true
    },
    "icc": false,
    "init": true,
    "insecure-registries": [
        "192.168.56.1:5000",
        "127.0.0.1:5000"
    ],
    "live-restore": false,
    "log-driver": "journald",
    "registry-mirrors": [
        "http://192.168.56.1:5000"
    ],
    "storage-driver": "overlay2",
    "userland-proxy": false
}
[vagrant@docker-host exp]$ docker version
Client:
 Version:           18.09.8
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        0dd43dd87f
 Built:             Wed Jul 17 17:40:31 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.8
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       0dd43dd
  Built:            Wed Jul 17 17:10:42 2019
  OS/Arch:          linux/amd64
  Experimental:     true

[vagrant@docker-host exp]$ cd /tmp/exp
[vagrant@docker-host exp]$ printf "FROM golang:alpine\nRUN echo 'test'" > Dockerfile
[vagrant@docker-host exp]$ docker build -t test .
[+] Building 0.1s (4/5)
 => [internal] load build definition from Dockerfile                                                                                                                          0.0s
 => => transferring dockerfile: 135B                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                               0.0s
 => ERROR [internal] load metadata for docker.io/library/golang:alpine                                                                                                        0.0s
 => ERROR [1/2] FROM docker.io/library/golang:alpine                                                                                                                          0.0s
 => => resolve docker.io/library/golang:alpine                                                                                                                                0.0s
------
 > [internal] load metadata for docker.io/library/golang:alpine:
------
------
 > [1/2] FROM docker.io/library/golang:alpine:
------
failed to do request: Head https://192.168.56.1:5000/v2/library/golang/manifests/alpine: http: server gave HTTP response to HTTPS client

Docker 19

[vagrant@docker-host exp]$ docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
[vagrant@docker-host exp]$ docker build -t test .
[+] Building 0.2s (4/5)
 => [internal] load build definition from Dockerfile                                                                                                                          0.1s
 => => transferring dockerfile: 95B                                                                                                                                           0.1s
 => [internal] load .dockerignore                                                                                                                                             0.1s
 => => transferring context: 2B                                                                                                                                               0.0s
 => ERROR [internal] load metadata for docker.io/library/golang:alpine                                                                                                        0.0s
 => ERROR [1/2] FROM docker.io/library/golang:alpine                                                                                                                          0.0s
 => => resolve docker.io/library/golang:alpine                                                                                                                                0.0s
------
 > [internal] load metadata for docker.io/library/golang:alpine:
------
------
 > [1/2] FROM docker.io/library/golang:alpine:
------
failed to solve with frontend dockerfile.v0: failed to build LLB: failed to load cache key: failed to do request: Head https://192.168.56.1:5000/v2/library/golang/manifests/alpine: http: server gave HTTP response to HTTPS client

Disable Buildkit

[vagrant@docker-host exp]$ cat /etc/docker/daemon.json
{
    "bip": "172.17.0.1/16",
    "debug": false,
    "dns": [
        "8.8.8.8",
        "10.80.1.10",
        "10.80.1.11"
    ],
    "experimental": true,
    "icc": false,
    "init": true,
    "insecure-registries": [
        "192.168.56.1:5000",
        "127.0.0.1:5000"
    ],
    "live-restore": false,
    "log-driver": "journald",
    "registry-mirrors": [
        "http://192.168.56.1:5000"
    ],
    "storage-driver": "overlay2",
    "userland-proxy": false
}
[vagrant@docker-host exp]$ sudo systemctl restart docker
[vagrant@docker-host exp]$ docker build -t test .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM golang:alpine
alpine: Pulling from library/golang
89d9c30c1d48: Already exists
8ef94372a977: Extracting [================>                                  ]   98.3kB/301.7kB
1ec62c064901: Download complete
339073ee4259: Downloading [=>                                                 ]  3.284MB/126.9MB
df12825c8c18: Download complete
eroji commented 4 years ago

Same exact issue. Using local Harbor repo. Reviewing logs on Harbor side shows the following. It seems like enabling BuildKit causes it to make some non-supported calls to the registry that leads to the failure...

2019/12/31 17:49:09 [D] [server.go:2774] |   10.1.131.206| 401 |   4.089799ms|   match| GET      /v2/   r:/v2/*
2019/12/31 17:49:10 [D] [server.go:2774] |   10.1.131.206| 200 |  24.498045ms|   match| GET      /service/token   r:/service/token
2019/12/31 17:49:10 [D] [server.go:2774] |   10.1.131.206| 200 |   4.120089ms|   match| HEAD     /v2/   r:/v2/*
2019/12/31 17:49:11 [D] [server.go:2774] |    10.1.131.33| 200 |   1.799942ms|   match| GET      /api/ping   r:/api/ping
2019/12/31 17:49:14 [D] [server.go:2774] |   10.1.131.206| 401 |   4.211535ms|   match| GET      /v2/   r:/v2/*
2019/12/31 17:49:14 [D] [server.go:2774] |   10.1.131.206| 200 |  64.820175ms|   match| GET      /service/token   r:/service/token
2019/12/31 17:49:14 [D] [server.go:2774] |   10.1.131.206| 200 |   8.106133ms|   match| GET      /v2/   r:/v2/*
2019/12/31 17:49:15 [D] [server.go:2774] |   10.1.131.206| 401 |    3.63903ms|   match| HEAD     /v2/library/openjdk/manifests/8u232-jre-stretch   r:/v2/*
2019/12/31 17:49:15 [C] [panic.go:522] the request url is  /service/token
2019/12/31 17:49:15 [C] [panic.go:522] Handler crashed with error '_xsrf' argument missing from POST
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/panic.go:522
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:80
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:164
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/controller.go:649
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/router.go:788
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:2774
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:1878
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/asm_amd64.s:1337
2019/12/31 17:49:15 [D] [server.go:2774] |   10.1.131.206| 401 |   4.155078ms|   match| HEAD     /v2/library/openjdk/manifests/8u232-jre-stretch   r:/v2/*
2019/12/31 17:49:15 [C] [panic.go:522] the request url is  /service/token
2019/12/31 17:49:15 [C] [panic.go:522] Handler crashed with error '_xsrf' argument missing from POST
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/panic.go:522
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:80
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/context/context.go:164
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/controller.go:649
2019/12/31 17:49:15 [C] [panic.go:522] /harbor/src/vendor/github.com/astaxie/beego/router.go:788
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:2774
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/net/http/server.go:1878
2019/12/31 17:49:15 [C] [panic.go:522] /usr/local/go/src/runtime/asm_amd64.s:1337
matevzmihalic commented 4 years ago

I have the same problem when I use GCR as a registry mirror. After I went digging through the code I concluded that the problem is in Moby and not in BuildKit.

This already merged PR fixes the problem for me, so it should be resolved with the next release of Docker.

tonistiigi commented 4 years ago

https://github.com/moby/buildkit/pull/1397

paddy-hack commented 1 year ago

Just in case someone hits the same.

TL;DR: BuildKit does not respect HTTP Docker Registry Mirrors (if it gets a bad reply to its HTTPS request)

I'm seeing this with

$ dpkg-query -W | grep docker
docker-buildx-plugin    0.10.2-1~debian.11~bullseye
docker-ce   5:23.0.1-1~debian.11~bullseye
docker-ce-cli   5:23.0.1-1~debian.11~bullseye
docker-compose-plugin   2.16.0-1~debian.11~bullseye

and

{
  "registry-mirrors": [
    "http://docker-registry.mirror.internal"
  ]
}

in /etc/docker/daemon.json.

Without BuildKit, a docker build hits the registry. With BuildKit it doesn't and goes to docker.io instead.

The docker.log shows

time="2023-02-15T10:41:50.584200726+09:00" level=info msg="trying next host" error="failed to do request: Head \"https://docker-registry.mirror.internal/v2/library/hello-world/manifests/latest?ns=docker.io\": x509: certificate is valid for ingress.local, not docker-registry.mirror.internal" host=docker-registry.mirror.internal

The registry mirror is hosted on a Kubernetes cluster and its Ingress only configures HTTP. Hence, any HTTPS request falls through to the cluster's default backend. That uses a self-signed certificate from Acme Co for common name of Kubernetes Ingress Controller Fake Certificate.

Adding the registry mirror to the list of insecure registries does not change anything.

I'm off setting up HTTPS support for our registry mirror but really think BuildKit should take note of the http:// on the registry mirror's URL and behave accordingly.

paddy-hack commented 1 year ago

FTR, our registry mirror now does HTTPS (with a certificate signed with our self-signed internal root CA certificate). The registry mirror permanently redirects incoming HTTP traffic with a HTTP Status 308 to the HTTPS URL. That made things work for me.

It works with http:// as well as https:// URLs for the registry mirror in /etc/docker/daemon.json. This should make transitioning a lot easier as long as all clients follow redirects and trust the (internal root CA) certificate. To be on the safe side (and for clarity's sake), I'll be changing URLs throughout our setup ... :sweat_drops: