Trivy: Image scanning doesn't appear to use local images

[mook-as]

It looks like when we scan images, we're not scanning the local images; this means that:

• If the local image has an issue that's been fixed in the registry, the user won't know about it
• I have no idea if it's downloading extra data to scan.
• If the local image is not in the registry, scanning fails.

Steps to reproduce:

• Create a dummy Dockerfile.
• Build an image with that Dockerfile, using the name mookas/junk:latest (that image is set to private in Docker Hub)
• Attempt to scan that image.

Expected results:

• Scan succeeds with no vulnerabilities (depending on what you're doing in your Dockerfile)

Actual results:

• Error when attempting to scan:

  
  FATAL scan error: unable to initialize a scanner: unable to initialize a docker scanner: 3 errors occurred:
  * unable to inspect the image (index.docker.io/mookas/junk:latest): Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  * unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
  * GET https://index.docker.io/v2/mookas/junk/manifests/latest: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:mookas/junk Type:repository]]

exit status 1

[jandubois]

Looking through Trivy issues, it looks like you can scan local images by name, but not by image id, and we seem to be scanning by id: https://github.com/aquasecurity/trivy/issues/1506

Furthermore, there is conflicting information about scanning local images with containerd. One issue claims it should work (since 0.29?): https://github.com/aquasecurity/trivy/issues/851

Another issue claims it doesn't actually work (on EKS): https://github.com/aquasecurity/trivy/issues/2540

Things to investigate:

• can we do local scanning by image id with some additional configuration?
• can we do local scanning by name using docker?
• the issue above talks about a podman socket; do we need podman installed as well?

[pmengelbert]

related: https://github.com/aquasecurity/trivy/issues/3048 I plan on submitting a PR to fix this in trivy, most likely early next week.

[pmengelbert]

@jandubois

Things to investigate:

• can we do local scanning by image id with some additional configuration?
• can we do local scanning by name using docker?
• the issue above talks about a podman socket; do we need podman installed as well?

Local scanning by ImageID can't be done with trivy at present. Trivy uses containerd's GetImage() call, which only searches by name. Trivy needs to use ListImages() instead, which allows for filtering by, among other things, ImageID.

Searching the containerd store is complicated by a few factors. Trivy mangles the image names in a way that messes things up. Trivy also only searches the default namespace in containerd. It's hard-coded in their source code but should be resolved soon https://github.com/aquasecurity/trivy/pull/3060 .

Trivy tends to support things in the Docker store a little better. There's only one namespace, so it's a little simpler.

A podman socket isn't needed. When trying to resolve an image reference, trivy goes through 1) the local docker store, 2) the local podman store, 3) the local containerd store, and 4) by looking it up in a registry. The error message about podman is just a side-effect of that resolution process.

[pmengelbert]

• Build an image with that Dockerfile, using the name mookas/junk:latest (that image is set to private in Docker Hub)

@mook-as internally, trivy will prefix mookas/junk with docker.io/, resulting in docker.io/mookas/junk. It won't find this image locally because you have it stored locally as mookas/junk! This is an issue in Trivy and I hope it gets resolved soon.

[jandubois]

related: aquasecurity/trivy#3048 I plan on submitting a PR to fix this in trivy, most likely early next week.

Thank you! Also for all the information in the following comment!

Please keep us updated with any progress, so we can try to improve local image scanning in Rancher Desktop, even if just for a subset of use cases.

[mirekphd]

trivy will prefix mookas/junk with docker.io/, resulting in docker.io/mookas/junk. It won't find this image locally because you have it stored locally as mookas/junk

If that were the case, then simply docker-tagging it with the hard-coded prefix (and the correct prefix would be probably registry.hub.docker.com rather than docker.io) would provide a workaround for this issue. Sadly, it does not help here (see below).

$ docker tag mirekphd/ml-cache:20230731 registry.hub.docker.com/mirekphd/ml-cache:20230731
$ ./scan-with-dockerized-trivy.sh registry.hub.docker.com/mirekphd/ml-cache:20230731
2023-08-05T14:27:14.686Z    DEBUG   Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2023-08-05T14:27:14.687Z    DEBUG   Ignore statuses {"statuses": null}
2023-08-05T14:27:14.708Z    DEBUG   cache dir:  /tmp/trivy/tmp/.cache
2023-08-05T14:27:14.708Z    DEBUG   DB update was skipped because the local DB is the latest
2023-08-05T14:27:14.708Z    DEBUG   DB Schema: 2, UpdatedAt: 2023-08-05 12:08:41.804819202 +0000 UTC, NextUpdate: 2023-08-05 18:08:41.804818802 +0000 UTC, DownloadedAt: 2023-08-05 13:37:02.676917362 +0000 UTC
2023-08-05T14:27:14.708Z    INFO    Vulnerability scanning is enabled
2023-08-05T14:27:14.708Z    DEBUG   Vulnerability type:  [os library]
2023-08-05T14:27:15.461Z    FATAL   image scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.Run
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:426
  - scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:268
  - unable to initialize a scanner:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
        /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:680
  - unable to initialize a docker scanner:
    github.com/aquasecurity/trivy/pkg/commands/artifact.imageStandaloneScanner
        /home/runner/work/trivy/trivy/pkg/commands/artifact/scanner.go:17
  - 4 errors occurred:
    * unable to inspect the image (registry.hub.docker.com/mirekphd/ml-cache:20230731): permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/registry.hub.docker.com/mirekphd/ml-cache:20230731/json": dial unix /var/run/docker.sock: connect: permission denied
    * containerd socket not found: /run/containerd/containerd.sock
    * unable to initialize Podman client: no podman socket found: stat podman/podman.sock: no such file or directory
    * GET https://registry.hub.docker.com/v2/mirekphd/ml-cache/manifests/20230731: MANIFEST_UNKNOWN: manifest unknown; unknown tag=20230731