search

slimtoolkit / slim

Slim(toolkit): Don't change anything in your container image and minify it by up to 30x (and for compiled languages even more) making it secure too! (free and open source)
Apache License 2.0
18.78k stars 700 forks source link

dockerutil.BuildEmptyImage() fails with bad API version #646

Open jimcottrell opened 4 months ago

jimcottrell commented 4 months ago

I've got a CI/CD pipeline that's run occasionally when we rebuild a image and create an optimized version with Slim. Without anything having changed that I'm aware of (same Slim version, same pipeline runner), this process has started failing with:

dockerutil.BuildEmptyImage: dockerapi.BuildImage() error = API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

I thought this might be related to #637, but it seems to be a separate issue, and the same thing happens whether or not I try setting --crt-api-version. Having updated everything to the latest versions, the problem continues. The issue seems to be specific to this environment, as a local build works.

Steps to Reproduce the Problem

1. slim build --http-probe-off --continue-after=enter alpine *(trivial reproduction; of course not our actual image)* ``` cmd=build state=started cmd=build info=params target.image='alpine' continue.mode='enter' rt.as.user='true' keep.perms='true' tags='' image-build-engine='internal' target.type='image' cmd=build state=image.inspection.start cmd=build info=image id='sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd' size.bytes='0' size.human='0 B' cmd=build info=image.stack index='0' name='alpine:latest' id='sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd' cmd=build state=image.inspection.done cmd=build state=container.inspection.start time="2024-02-26T18:12:39Z" level=error msg="dockerutil.BuildEmptyImage: dockerapi.BuildImage() error = API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\n / output: " time="2024-02-26T18:12:39Z" level=fatal msg="slim: failure" error="API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\n" stack="goroutine 1 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngithub.com/slimtoolkit/slim/pkg/util/errutil.FailOn({0x231e3e0, 0xc0007e62a0})\n\tgithub.com/slimtoolkit/slim/pkg/util/errutil/errutil.go:32 +0x4b\ngithub.com/slimtoolkit/slim/pkg/app/master/inspectors/container.(*Inspector).RunContainer(0xc000464b00)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/inspectors/container/container_inspector.go:430 +0x77a\ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.OnCommand(_, _, {_, _}, _, {_, _}, {_, _}, {0x0, ...}, ...)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/handler.go:1132 +0x4632\[ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.glob..func1(0xc0003d35c0)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/cli.go:760](http://ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.glob..func1(0xc0003d35c0)/n/tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/cli.go:760) +0x5d51\[ngithub.com/urfave/cli/v2.(*Command](http://ngithub.com/urfave/cli/v2.(*Command)).Run(0x3396180, 0xc0000af480)\n\[tgithub.com/urfave/cli/v2@v2.3.0/command.go:163](http://tgithub.com/urfave/cli/v2@v2.3.0/command.go:163) +0x583\[ngithub.com/urfave/cli/v2.(*App](http://ngithub.com/urfave/cli/v2.(*App)).RunContext(0xc000266d00, {0x233bb88?, 0x340d500}, {0xc00012e000, 0x7, 0x7})\n\[tgithub.com/urfave/cli/v2@v2.3.0/app.go:313](http://tgithub.com/urfave/cli/v2@v2.3.0/app.go:313) +0xaa5\[ngithub.com/urfave/cli/v2.(*App](http://ngithub.com/urfave/cli/v2.(*App)).Run(...)\n\[tgithub.com/urfave/cli/v2@v2.3.0/app.go:224\ngithub.com/slimtoolkit/slim/pkg/app/master.Run()\n\tgithub.com/slimtoolkit/slim/pkg/app/master/app.go:15](http://tgithub.com/urfave/cli/v2@v2.3.0/app.go:224/ngithub.com/slimtoolkit/slim/pkg/app/master.Run()/n/tgithub.com/slimtoolkit/slim/pkg/app/master/app.go:15) +0x45\nmain.main()\n\[tgithub.com/slimtoolkit/slim/cmd/slim/main.go:15](http://tgithub.com/slimtoolkit/slim/cmd/slim/main.go:15) +0x187\n" version="linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM" ``` --- Specifications ================= - Version: 1.40.11 - Platform: linux/amd64 (self-hosted Bitbucket pipeline runner, which is configured for privileged Docker access so Slim can run) - Output of `slim version`: ``` cmd=version info=app container='true' dsimage='false' location='/opt/atlassian/pipelines/agent/build/dist_linux' current='1.40.11' verdict='you have the latest version' cmd='version' version='linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM' cmd=version info=host cmd='version' osname='Ubuntu 22.04.3 LTS' osbuild='' version='#18~22.04.1-Ubuntu SMP Wed Jan 10 22:54:16 UTC 2024' release='6.2.0-1018-aws' sysname='Linux' cmd=version info=docker server.version='25.0.3' architecture='x86_64' cmd='version' name='a640edfcc04d' kernel.version='6.2.0-1018-aws' operating.system='Alpine Linux v3.19 (containerized)' ostype='linux' cmd=version info=dclient build.time='2024-02-06T21:13:08.000000000+00:00' git.commit='f417435' cmd='version' api.version='1.44' min.api.version='1.24' app='slim' message='GitHub Discussions' info='https://github.com/slimtoolkit/slim/discussions' app='slim' message='Join the CNCF Slack channel to ask questions or to share your feedback' info='https://cloud-native.slack.com/archives/C059QP1RH1S' app='slim' message='Join the Discord server to ask questions or to share your feedback' info='https://discord.gg/9tDyxYS' app='slim' message='Join the Gitter channel to ask questions or to share your feedback' info='https://gitter.im/docker-slim/community' ```
kcq commented 4 months ago

Interesting... need to investigate/repro. Either way, the next release should address this. Check back a bit later or follow me on Github or Twitter to be notified when the new release is out.

jimcottrell commented 4 months ago

Sounds good, thanks. I'll see how the next version works out. We're still having some build issues, but it looks like if we set a DOCKER_API_VERSION environment variable, we can at least get past this error.

jimcottrell commented 4 months ago

For reference, other versions may work (1.44 did not), but we seem to be back in business with

export DOCKER_API_VERSION='1.25'

in our pipeline

kcq commented 2 months ago

@jimcottrell this might work as-is https://github.com/mintoolkit/mint/releases/tag/1.41.1 let me know if you still have problems there

jimcottrell commented 2 months ago

I've tried with v1.41.1 and no explicit DOCKER_API_VERSION and I get that same error: client version 1.12 is too old

kcq commented 2 months ago

@jimcottrell which version of Docker do you have installed? Are you on Mac or Linux? If you are on Linux are you using Docker Desktop or only Docker Engine? If you are on a Mac is it an AMD or an M1/M2? Trying to repro... more info on your environment will help

sammcj commented 1 month ago

Getting this on Fedora 39 with the latest stable docker-ce and slim installed.

slim build --target nvapi --tag slim --http-probe=false --dockerfile Dockerfile --dockerfile-context .
...
cmd=build info=build.error status='standard.image.build.error' value='API /build requires version 1.25, requested version 1.24 is insufficient'
cmd=build state=exited code=33554435 version=linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM location=/usr/local/bin
dnf info docker-ce-26.1.1-1.fc39.x86_64
Last metadata expiration check: 4:53:24 ago on Sun 05 May 2024 09:04:18.
Installed Packages
Name         : docker-ce
Epoch        : 3
Version      : 26.1.1
Release      : 1.fc39
Architecture : x86_64
Size         : 104 M
Source       : docker-ce-26.1.1-1.fc39.src.rpm
Repository   : @System
From repo    : docker-ce-stable
Summary      : The open-source application container engine
URL          : https://www.docker.com

Setting DOCKER_API_VERSION='1.25' makes no difference.

Seems to be a common problem looking at #637 and caused by #641

kcq commented 1 month ago

@sammcj do you get the same result with this version?

kcq commented 1 month ago

@sammcj Here's the Vagrant-based repro with Fedora 39 and Docker 26.1.1. You should be good with the release version in the repro config. The release version you shared in this issue is out of date.

jimcottrell commented 4 weeks ago

@kcq Sorry for the delay in responding. As noted before, this is happening only on a self-hosted Bitbucket pipeline runner (linux/amd64). When I run locally, I have no issue. To make Slim work in the pipeline environment, it has to run via the docker:dind image.

Host Docker

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:45 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker:dind

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:58 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.15
  GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Looking at this does seem to explain why I couldn't set it to use v1.44.

kcq commented 4 weeks ago

@kcq Sorry for the delay in responding. As noted before, this is happening only on a self-hosted Bitbucket pipeline runner (linux/amd64). When I run locally, I have no issue. To make Slim work in the pipeline environment, it has to run via the docker:dind image.

Host Docker

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:45 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker:dind

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:58 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.15
  GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Looking at this does seem to explain why I couldn't set it to use v1.44.

@jimcottrell Running with dind in Bitbucket runners makes it more complicated because each CI/CD vendor does the dind setup in slightly different ways, which requires different ways to connect to the (correct) Docker runtime. Some environments don't expose the Docker unix socket. Other environments need certs to connect to the Docker runtime. Some have more advanced network configurations that prevents direct container communication, so the --sensor-ipc-mode and --sensor-ipc-endpoint flags need to be used. The RUNNING CONTAINERIZED README section covers a few examples

To troubleshoot your setup we need two things to get started:

  1. How you invoke the slim command in your dind setup (if it's slim build --http-probe-off --continue-after=enter alpine then it needs to be updated to point to the dind Docker runtime. optionally you can try using the containerized version of the app, but that's not required).
  2. Your Bitbucket runner config that defines how the dind magic is configured.
jimcottrell commented 3 weeks ago

Here's a simplified version of our bitbucket-pipelines.yml that fails with client version 1.12 is too old:

image:
  name: atlassian/default-image:4
definitions:
  services:
    privileged-docker:
      type: docker
      image:
        name: docker:dind
pipelines:
  custom:
    test:
      - step:
          runs-on:
            - linux
            - self.hosted
          name: Build Optimized Image
          services:
            - privileged-docker
          script:
            - curl https://downloads.dockerslim.com/releases/1.40.11/dist_linux.tar.gz -o slim.tar.gz
            - tar xvf slim.tar.gz
            #- export DOCKER_API_VERSION='1.25' # Temporary, see https://github.com/slimtoolkit/slim/issues/646
            - ./dist_linux/slim build --continue-after=enter alpine

I tried it with --in-container=true which didn't make any difference. What needs to be updated to point to the dind Docker runtime? Apart from this one case, I'm not very familiar with dind. The DOCKER_HOST environment variable exists, but I don't believe the Docker UNIX socket is present.