search

moby / buildkit

concurrent, cache-efficient, and Dockerfile-agnostic builder toolkit
https://github.com/moby/moby/issues/34227
Apache License 2.0
8.04k stars 1.13k forks source link

--no-cache option is ignored by RUN #4535

Closed jgfoster closed 8 months ago

jgfoster commented 8 months ago

I'm finding that even RUN statements are showing as CACHED.

jfoster@JGF-MBP-14-2022 flutter % docker --version
Docker version 24.0.7, build afdd53b
jfoster@JGF-MBP-14-2022 flutter % docker build --no-cache -t theia-flutter .       
[+] Building 0.1s (17/18)                                                            docker:desktop-linux
 => [internal] load build definition from Dockerfile                                                 0.0s
 => => transferring dockerfile: 1.63kB                                                               0.0s
 => [internal] load .dockerignore                                                                    0.0s
 => => transferring context: 2B                                                                      0.0s
 => [internal] load metadata for docker.io/library/theia-common:latest                               0.0s
 => [internal] load build context                                                                    0.0s
 => => transferring context: 2B                                                                      0.0s
 => CACHED [ 1/14] FROM docker.io/library/theia-common                                               0.0s
 => [ 2/14] RUN echo "Hello, world!" > /tmp/hello.txt                                                0.1s
 => CACHED [ 3/14] RUN apt update && apt install -y libglu1-mesa                                     0.0s
 => CACHED [ 4/14] WORKDIR /opt                                                                      0.0s
 => CACHED [ 5/14] RUN wget https://foo.com/bar.txt                                                  0.0s
 => CACHED [ 6/14] RUN wget https://storage.googleapis.com/flutter_infra_release/releases/stable/li  0.0s
 => CACHED [ 7/14] RUN tar xf flutter* && rm flutter*.tar.xz                                         0.0s
 => CACHED [ 8/14] RUN   ln -s /opt/flutter/bin/dart /usr/local/bin/dart &&   ln -s /opt/flutter/bi  0.0s
 => CACHED [ 9/14] RUN chmod -R o+w /opt/flutter                                                     0.0s
 => CACHED [10/14] RUN wget https://github.com/Dart-Code/Dart-Code/releases/download/v3.80.0/dart-c  0.0s
 => CACHED [11/14] RUN wget https://github.com/Dart-Code/Flutter/releases/download/v3.80.0/flutter-  0.0s
 => ERROR [12/14] ADD dart-code-3.80.0.vsix /opt/theia/plugins/                                      0.0s
 => ERROR [13/14] ADD flutter-3.80.0.vsix /opt/theia/plugins/                                        0.0s

Editing the foo.com/bar.txt line still shows cached!

Copied from here.

thaJeztah commented 8 months ago

Can you provide;

jgfoster commented 8 months ago 
jfoster@JGF-MBP-14-2022 online-ide % docker version
Client:
 Cloud integration: v1.0.35+desktop.5
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:20 2023
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.26.1 (131620)
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.25
  GitCommit:        d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc:
  Version:          1.1.10
  GitCommit:        v1.1.10-0-g18a0cb0
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
jfoster@JGF-MBP-14-2022 online-ide % docker info
Client:
 Version:    24.0.7
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.0-desktop.2
    Path:     /Users/jfoster/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.3-desktop.2
    Path:     /Users/jfoster/.docker/cli-plugins/docker-compose
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.0
    Path:     /Users/jfoster/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.21
    Path:     /Users/jfoster/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  0.1
    Path:     /Users/jfoster/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v0.1.0-beta.10
    Path:     /Users/jfoster/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/jfoster/.docker/cli-plugins/docker-sbom
  scan: Docker Scan (Docker Inc.)
    Version:  v0.26.0
    Path:     /Users/jfoster/.docker/cli-plugins/docker-scan
  scout: Docker Scout (Docker Inc.)
    Version:  v1.2.0
    Path:     /Users/jfoster/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.5.11-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 7.663GiB
 Name: docker-desktop
 ID: 9e58b66d-b191-4373-a99b-4372b310808d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile
jfoster@JGF-MBP-14-2022 online-ide % docker buildx ls
NAME/NODE             DRIVER/ENDPOINT  STATUS   BUILDKIT             PLATFORMS
multi-arch-builder *  docker-container                               
  multi-arch-builder0 desktop-linux    inactive                      
default               docker                                         
  default             default          running  v0.11.7+d3e6c1360f6e linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
desktop-linux         docker                                         
  desktop-linux       desktop-linux    running  v0.11.7+d3e6c1360f6e linux/arm64, linux/amd64, linux/amd64/v2, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
[Dockerfile (common).txt](https://github.com/moby/buildkit/files/13864412/Dockerfile.common.txt)
[Dockerfile (flutter).txt](https://github.com/moby/buildkit/files/13864415/Dockerfile.flutter.txt)

jfoster@JGF-MBP-14-2022 online-ide %
jgfoster commented 8 months ago

Dockerfile (common).txt Dockerfile (flutter).txt

thaJeztah commented 8 months ago

Thanks! From that output, it looks like you're using a custom containerised builder, but it's not running. Do you know what version of BuildKit it uses?

thaJeztah commented 8 months ago

/cc @tonistiigi @crazy-max

tonistiigi commented 8 months ago

This looks like weird output for errored build. The "cached" there really means that the step did not run. The error happened in the child of that step so that's why step itself needs to be shown. Can you reproduce something similar without a build that errors.

Not sure what the better name would be there. Maybe "skipped".

jgfoster commented 8 months ago

@tonistiigi – I'm not familiar enough with Docker to know what is meant by "child" and "step." I've provided the Dockerfile and it is mostly a series of RUN commamds (is each RUN a "step"?). The error happens at line 22 with an ADD for a file that doesn't exist. The reason it doesn't exist is because the preceeding RUN wget ... was skipped ("CACHED"). While I could remove the ADD to avoid the error, I don't understand how an ADD command is a "child" of the prior "RUN" command, and I certainly wouldn't expect that removing the ADD would cause the earlier RUN to not be skipped.

@thaJeztah – I don't know what is mean by "a custom containerised builder" and I don't know anything about BuildKit. I've provided the Dockerfile for both images. I'm happy to follow instructions to get the needed info.

Note that this script ran fine a month ago.

tonistiigi commented 8 months ago

The error happens at line 22 with an ADD for a file that doesn't exist. The reason it doesn't exist is because the preceeding RUN wget ... was skipped

If the failing line is 

ADD dart-code-3.80.0.vsix /opt/theia/plugins/

And it fails because it doesn't exist it is because the file does not exist in your build context. The wget is not generating that file.

I don't understand how an ADD command is a "child" of the prior "RUN" command, and I certainly wouldn't expect that removing the ADD would cause the earlier RUN to not be skipped.

The above ADD command directly depends on two previous steps (you can call them parents), one is transfer of build context that provides the dart-code-3.80.0.vsix file and another is the previous RUN command that provides the destination directory /opt/theia/plugins/.

BuildKit can detect that ADD dart-code-3.80.0.vsix will 100% fail because the file does not exist and it doesn't need to execute the previous RUN wget command to know that. In the output, RUN wget is shown as "cached" to signify that the step never ran, it would be better if it would be shown as "skipped".

jgfoster commented 8 months ago

I now see that I was confusing my source environment with the target environment. I used to do the wget in my source environment and then ADD to the target environment. I wanted to switch to downloading directly into the target and should simply remove the ADD. My bad; thanks for everyone's patience and help.

thaJeztah commented 8 months ago

This looks like weird output for errored build. The "cached" there really means that the step did not run. The error happened in the child of that step so that's why step itself needs to be shown. Can you reproduce something similar without a build that errors.

Ah, interesting, I didn't know it would check the COPY / ADD before executing RUN. So in this case;

Not sure what the better name would be there. Maybe "skipped".

I guess the confusing bit is also that the RUN steps show before the ADD checks, so the presentation shows in which order they would be executed if validation would pass, but effectively there's two passes;

Unless we want to show both in the output, and with in mind that the RUN is never executed in this case, perhaps CANCELED would make sense as wording for this WDYT @tonistiigi ?

thiner commented 6 months ago

This looks like weird output for errored build. The "cached" there really means that the step did not run. The error happened in the child of that step so that's why step itself needs to be shown. Can you reproduce something similar without a build that errors.

Ah, interesting, I didn't know it would check the COPY / ADD before executing RUN. So in this case;

  • it validates the syntax
  • it checks / validates the ADD / COPY
  • determines it's invalid (file missing)
  • terminates the build, as executing the build would always fail

Not sure what the better name would be there. Maybe "skipped".

I guess the confusing bit is also that the RUN steps show before the ADD checks, so the presentation shows in which order they would be executed if validation would pass, but effectively there's two passes;

  • validation (RUN syntax is correct; source file for ADD / COPY are present)
  • execution (RUN -> ADD / COPY)

Unless we want to show both in the output, and with in mind that the RUN is never executed in this case, perhaps CANCELED would make sense as wording for this WDYT @tonistiigi ?

This is a bug, isn't it? My Dockerfile has certain flow that it removes files by deleting the whole folder in RUN command, and then copy build-out files into it again by COPY command. The validation logic fails my docker build process currently.