ResolveImageConfig takes 70.1s to complete

monkeyWie commented 1 year ago

I enabled BUILDKIT, after a successful build image, the first step of building again does not use caching and is very slow. resolve image config takes 70 seconds!

console

[internal] load .dockerignore
#2 transferring context: 125B done
#2 DONE 0.0s

#3 resolve image config for docker.io/docker/dockerfile:1.5.2@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14
#3 DONE 70.1s

#4 docker-image://docker.io/docker/dockerfile:1.5.2@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14
#4 CACHED

#5 [internal] load metadata for docker.io/library/openjdk:8-jdk-buster
#5 DONE 0.0s

#6 [internal] load metadata for docker.io/library/maven:3.6.3-openjdk-8
#6 DONE 0.0s

#7 [builder 1/6] FROM docker.io/library/maven:3.6.3-openjdk-8
#7 DONE 0.0s

Dockerfile

# syntax=docker/dockerfile:1.5.2@sha256:39b85bbfa7536a5feceb7372a0817649ecb2724562a38360f4d6a7782a409b14
FROM maven:3.6.3-openjdk-8 AS builder
WORKDIR /build
COPY . .
RUN --mount=target=/root/.m2,type=cache \
    mvn package -Dmaven.test.skip=true

FROM openjdk:8-jdk-buster
WORKDIR /server
COPY --from=builder /build/target/app.jar ./app.jar
CMD ["java","-jar","app.jar"]

docker version

Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:47:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:47:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.18
  GitCommit:        2456e983eb9e37e47538f59ea18f2043c9a73640
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

jedevc commented 1 year ago

What's the command you're using to build the image?

Can you reproduce, or was this a one-off failure?

monkeyWie commented 1 year ago

Hi @jedevc, the build command being used is DOCKER_BUILDKIT=1 docker build -t app-dev:latest ., and it seems that the issue is being replicated almost every time. From my perspective, the initial steps of the build process should use the cache, so network requests shouldn't really be necessary.

furkanmustafa commented 6 months ago

I believe this issue is same or related with the following;

and probably a few more... and it is there for a few (3-5) years now. It happens on Apline Linux and MacOS(alpine/docker on lima) for me.

Core of the problem (existence of network op)

trying to load metadata remotely for a locally available image for no apparent reason, each time. I expect absolutely ZERO network ops if the image is built once and all parts available locally.

Side problem (slow network op)

for some reason, only this "load metadata" problem is much slower than the consecutive operations. probably due to network/dns issues.

I could not find any way to disable remote metadata lookups.. trying to read related code;

Available workarounds

None of the following is a good solution

Use a local caching(pull-through) registry
Re-tagging your FROM images locally
disable buildkit

AnrDaemon commented 2 months ago

 => [php-fpm internal] load build definition from Dockerfile                                 0.0s
 => => transferring dockerfile: 1.30kB                                                       0.0s
 => [php-fpm internal] load metadata for docker.io/library/alpine:3.14                       0.0s <= Ok…
 => [php-fpm internal] load metadata for docker.io/library/composer:2.2.23                  30.5s <=== !!!
 => [php-fpm internal] load .dockerignore                                                    0.0s
 => => transferring context: 189B                                                            0.0s

$ docker --version Docker version 26.1.3, build b72abbb

The systems is a VM in a datacenter. On a 10G link with lightning fast DNS responses.

AnrDaemon commented 2 months ago

Following the advice in https://stackoverflow.com/a/70483395/1449366 and just pulled the problematic images manually. From now on, the builds are lightning fast. (And no, I did not re-tag them, just did docker pull ….)

moby / buildkit