search

goodwithtech / dockle

Container Image Linter for Security, Helping build the Best-Practice Docker Image, Easy to start
https://containers.goodwith.tech/
Apache License 2.0
2.75k stars 138 forks source link

dockle stopped working when updating docker to version 25 #256

Open tgquan67 opened 6 months ago

tgquan67 commented 6 months ago

Description

Since I updated docker engine to version 25, I cannot scan images built locally anymore, with or without buildkit. The image is built normally with a dockerfile

docker build . -t a:b --load

And scan with

dockle a:b

This happens on both my local machine with Ubuntu 20.04 and dockle 0.4.11 and my Jenkins system with Ubuntu 20.04 and dockle 0.4.14. I already tried setting DOCKLE_HOST but that did not solve the problem.

What did you expect to happen? The image is scanned.

What happened instead?

2024-03-05T15:55:55.182+0200    FATAL   unable to initialize a image struct: failed to initialize source: reading manifest b in docker.io/library/a: requested access to the resource is denied

Output of run with -debug:

2024-03-05T15:56:01.945+0200    DEBUG   There is no .dockleignore file
2024-03-05T15:56:01.945+0200    DEBUG   Skipped update confirmation
2024-03-05T15:56:01.945+0200    DEBUG   Start assessments...
2024-03-05T15:56:03.669+0200    FATAL   unable to initialize a image struct:
    github.com/goodwithtech/deckoder/extractor/docker.newDockerExtractor
        /home/runner/go/pkg/mod/github.com/goodwithtech/deckoder@v0.0.3/extractor/docker/docker.go:73
  - failed to initialize source:
    github.com/goodwithtech/deckoder/extractor/image.NewImage
        /home/runner/go/pkg/mod/github.com/goodwithtech/deckoder@v0.0.3/extractor/image/image.go:86
  - reading manifest b in docker.io/library/a: requested access to the resource is denied

Output of dockle -v:

dockle version 0.4.11

Additional details (base image name, container registry info...): Sample dockerfile used

FROM python:3.8
COPY Pipfile Pipfile.lock .
RUN pip install pipenv && \
    pipenv sync
COPY code.txt .
lamroger-nava commented 4 months ago

I'm having the same issue - did you figure out the problem @tgquan67 ?

edit: I figured it out - the runner / docker was out of space. I run this before checkout

- name: Delete huge unnecessary tools folder
        run: rm -rf /opt/hostedtoolcache
tgquan67 commented 4 months ago

As far as I can see, the problem seems to have been solved with docker 26, not sure what docker did though.

BertelBB commented 2 months ago

Also experiencing this.

Docker version 26.1.3, build b72abbb

Seems that Dockle is no longer able to scan local images. Always tries to pull from DockerHub, even if no user is set in the image tag. E.g. app-name:latest. I think historically, Docker would assume images without a registry or docker-hub-user/ in the image name were local images. Now it always tries to pull...

phyzical commented 2 months ago

also just started running into this due to i think the docker version being updated on github runnners base image? i dont have this issue on custom runners atm

edit: yeah looks like the ubuntu-latest bumped the docker server version from 24 to 26 last night https://github.com/actions/runner-images/blame/main/images/ubuntu/Ubuntu2204-Readme.md#L82

Why its behaving this why idk... maybe something buildkit related now that 26 tries to push it harder?

BertelBB commented 2 months ago

I managed to solve this by using the --load flag when I build the container image. I could not get it to work when using goodwithtech/dockle-action@v0.1.2. Not sure if that is because it uses dockle@0.4.13 or something else. Instead of using the action, I simply run Dockle as a container.

This is what I am doing in a nutshell:

steps:
- name: Checkout
  uses: actions/checkout@v4

- name: Setup Docker buildx
  uses: docker/setup-buildx-action@v3

- name: Docker build
  run: |
    docker buildx build -t app:latest --load ./src

- name: Lint container using Dockle
  uses: docker://goodwithtech/dockle:v0.4.14
  with:
    args: app:latest
tgquan67 commented 2 months ago

In @BertelBB's case, it's not related to dockle, it's just how docker buildkit works (which was enabled by default since docker 23). Basically when you build with buildkit, the result image will remain in cache and will not be exported to docker daemon (where dockle tries to find the image) unless you explicitly tell it so with --load. It's mostly related to the builder you use, but I have seen other people who use almost the same builder as mine but doesn't need --load, so the behavior is a bit inconsistent, and I haven't been able to track down the exact difference.

BertelBB commented 2 months ago

In @BertelBB's case, it's not related to dockle, it's just how docker buildkit works (which was enabled by default since docker 23). Basically when you build with buildkit, the result image will remain in cache and will not be exported to docker daemon (where dockle tries to find the image) unless you explicitly tell it so with --load. It's mostly related to the builder you use, but I have seen other people who use almost the same builder as mine but doesn't need --load, so the behavior is a bit inconsistent, and I haven't been able to track down the exact difference.

EDIT: I spoke too soon in my previous version of this comment. I am able to replicate this locally when using dockle@v0.4.13, even if I build using the --load flag.

In GH Actions, I had to use the --load flag and dockle@v0.4.14. I tried both docker://goodwithtech/dockle:v0.4.13 and goodwithtech/dockle-action@v0.1.2 (latest action version, targets dockle@v0.4.13) and it failed in both cases.

tgquan67 commented 2 months ago

@BertelBB can you confirm that after you build, you can find your new image in the output of docker images?

tgquan67 commented 2 months ago

Update: I can confirm that docker 26.1 is not working with dockle again:

$ docker images
REPOSITORY      TAG               IMAGE ID       CREATED              SIZE
a               b                 a49f44b85bbb   About a minute ago   77.9MB
moby/buildkit   buildx-stable-1   480495983c47   4 months ago         172MB

$ dockle a:b
2024-06-14T14:15:53.212+0300    FATAL   unable to initialize a image struct: failed to initialize source: reading manifest b in docker.io/library/a: requested access to the resource is denied

$ dockle -v
dockle version 0.4.11

$ docker --version
Docker version 26.1.3, build b72abbb

It worked on docker 24.0.8 but not 25, and iirc it also worked on docker 26.0. Try to downgrade docker to version 26.0.2 and test again.

BertelBB commented 2 months ago

I see the image when I do docker images, regardless of if I use --load flag or not.

$ docker --version
Docker version 26.1.4, build 5650f9b

$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

# Build without --load flag
$ docker buildx build --tag sample-service:latest ./src

$ docker images
REPOSITORY       TAG       IMAGE ID       CREATED         SIZE
sample-service   latest    3bb79129cf96   7 seconds ago   204MB

$ dockle --version
dockle version 0.4.14

# Scan with dockle@v0.4.14 on image built without --load flag
$ dockle sample-service:latest
WARN    - DKL-DI-0006: Avoid latest tag
        * Avoid 'latest' tag
INFO    - CIS-DI-0005: Enable Content trust for Docker
        * export DOCKER_CONTENT_TRUST=1 before docker pull/build
INFO    - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
        * not found HEALTHCHECK statement

# Downgrade Dockle to 0.4.13
$ curl -L -o dockle.deb https://github.com/goodwithtech/dockle/releases/download/v0.4.13/dockle_0.4.13_Linux-64bit.deb
$ sudo dpkg -i dockle.deb

$ dockle --version
dockle version 0.4.13

# Scan with dockle@v0.4.13 on image build without --load flag
$ dockle sample-service:latest
2024-06-14T11:49:31.163Z        FATAL   unable to initialize a image struct: failed to initialize source: reading manifest latest in docker.io/library/sample-service: requested access to the resource is denied

# Cleanup
$ docker system prune -fa

$ docker images
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

# Build with --load flag
$ docker buildx build --load --tag sample-service:latest ./src

$ docker images
REPOSITORY       TAG       IMAGE ID       CREATED         SIZE
sample-service   latest    15a358e41a3f   2 seconds ago   204MB

# Scan with dockle@v0.4.13 on image built with --load flag
$ dockle sample-service:latest
2024-06-14T11:52:14.955Z        FATAL   unable to initialize a image struct: failed to initialize source: reading manifest latest in docker.io/library/sample-service: requested access to the resource is denied

# Upgrade Dockle to v0.4.14
$ curl -L -o dockle.deb https://github.com/goodwithtech/dockle/releases/download/v0.4.14/dockle_0.4.14_Linux-64bit.deb
$ sudo dpkg -i dockle.deb

# dockle --version
$ dockle version 0.4.14

# Scan with dockle@v0.4.14 on image built with --load flag
$ dockle sample-service:latest
WARN    - DKL-DI-0006: Avoid latest tag
        * Avoid 'latest' tag
INFO    - CIS-DI-0005: Enable Content trust for Docker
        * export DOCKER_CONTENT_TRUST=1 before docker pull/build
INFO    - CIS-DI-0006: Add HEALTHCHECK instruction to the container image
        * not found HEALTHCHECK statement

As you can see from the above, using the --load flag doesn't seem to matter.

In GH Actions however, I had to use --load flag to get it to work...

tgquan67 commented 2 months ago

@BertelBB then I guess --load doesn't matter to this particular issue, just use it when you can't get your image to docker images. It seems like dockle version 0.4.14 is working for you, however on Ubuntu 20.04 the maximum version I can get from apt repo is 0.4.11, so I will try to install a newer version from deb package later

cpannwitz commented 2 months ago

Are there any updates, or working workarounds, to this situation?

Our Github Action looks a bit like this, and fails with the before mentioned error:

      - name: Checkout Code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: build local container
        uses: docker/build-push-action@v5
        with:
          tags: IMAGENAME:latest
          push: false
          load: true

      - name: Run dockle
        uses: goodwithtech/dockle-action@main
        env:
          DOCKLE_HOST: 'unix:///var/run/docker.sock'

        with:
          image: 'IMAGENAME:latest'
          format: 'list'
          exit-code: '1'
          exit-level: 'warn'
tgquan67 commented 2 months ago

@cpannwitz did you try @BertelBB's solution above? https://github.com/goodwithtech/dockle/issues/256#issuecomment-2166137411 and https://github.com/goodwithtech/dockle/issues/256#issuecomment-2167592655

cpannwitz commented 2 months ago

Yeah, tried it. Got it to work, essentially by not using the Github Action, but the Docker Container instead.

- name: Lint container using Dockle
        uses: docker://goodwithtech/dockle:v0.4.14
        with:
          args: '--exit-code 1 --exit-level warn --format list IMAGENAME:latest'
faramat commented 1 month ago

any ideas?

phyzical commented 1 month ago

using load resolved it for me too

But i also moved our pipeline over to using the build-push action + ghcr for intermediate storage instead of cli buildx + local refs.

so its possible im getting around the issue as its just pulling from the ghcr instead?