Unable to use Syft from GitLab CI/CD #1858

Closed kuwv closed 1 year ago

What happened:

GitLab utilizes shells of container images to run CI/CD jobs. The Syft container doesn't not provide any shells. While the distroless images do provide a focused base image for CLI images it prevents Syft from being used by all CI/CD platforms where it is intended.

What you expected to happen:

GitLab should be able to utilize a shell and perform any additional pre/post setup actions for Syft.

Steps to reproduce the issue:

Create a project with a pipeline on GitLab.com (or any CI/CD utilizing shells) and attempt to use syft:

image:
  name: docker.io/anchore/syft:v0.82.0
  entrypoint: ''

Anything else we need to know?: https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#override-the-entrypoint-of-an-image https://github.com/anchore/syft/issues/833 https://github.com/anchore/grype/issues/1335

Hi @kuwv - there is docker.io/anchore/syft:v0.82.0-debug variant which uses the distroless debug variant that has a shell included: https://github.com/anchore/syft/blob/main/Dockerfile.debug

@westonsteimel Thanks, it seems that this image had once supported shells and then was removed at some point?

Yes, I believe it was unintentional. The non-debug image was never meant to have a shell, but I think the original PR that split it out into two images accidentally used the distroless -debug for both and then eventually it was caught and fixed

Hmm, actually it looks like it still is using the debug variant, so I'm not sure

@spiffcs , any ideas here?

Hmm, actually it looks like it still is using the debug variant, so I'm not sure

Nevermind, it uses it as the build stage and pulls the certificates from it into the final layer, but the final layer is FROM scratch

Yeah, unfortunately no joy:

Status: Downloaded newer image for anchore/syft:v0.82.0-debug
docker: Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown.

For the distroless debug image the shell appears to be at /busybox/sh

Doesn't seem to work:

/busybox/sh: line 1: $
                      ??v
                         ??v
                            1??_R?: not found
/busybox/sh: line 1: ELF: not found
/busybox/sh: line 1: $
f.?@?P?QH=P?Qt?H??t: not found
/busybox/sh: line 2: syntax error: unexpected word (expecting ")")

It looks like /busybox/sh should still be compatible after taking a look at these documents: https://docs.gitlab.com/runner/executors/docker.html#configure-images-and-services

/busybox/sh: line 1: ELF: not found

This was mentioned by @westonsteimel regarding the above: https://stackoverflow.com/questions/57446579/executable-says-line-1-elf-not-found-when-starts

Is there an architecture incompatibility here that could be causing the above error?

@spiffcs my platform is mac/intel. That image could very well be using the local ELF path for whatever. I'll try to use it from the pipeline.

@westonsteimel Seems like the GitLab CI doesn't find that path:

Using docker image sha256:3a06f3c0f22f972de0c947d633bac3392e757abbc108cf104fa9922d6278bed3 for docker.io/anchore/syft:v0.82.0-debug with digest anchore/syft@sha256:bc395b0b9df0d5e1c7d1017a570dde1f0579584bc9bbb1c91800a21069bb6d94 ...
/busybox/sh: can't open 'sh': No such file or directory

  image:
    name: docker.io/anchore/syft:v0.82.0-debug
    entrypoint: ['/busybox/sh']

I was however able to shell into the image after a brew upgrade:

docker run --entrypoint /busybox/sh -it docker.io/anchore/syft:v0.82.0-debug
/tmp #

That's weird - doing some research I found: https://gitlab.com/gitlab-org/gitlab-runner/-/blob/3b214797b045fd0e9e5f834e12d9355314e1e27f/shells/bash.go#L16

^ Based on the docker command you ran, the entrypoint path, and the above script it should be able to detect ``/busybox/sh

I did a docker inspect anchore/syft@sha256:bc395b0b9df0d5e1c7d1017a570dde1f0579584bc9bbb1c91800a21069bb6d94

Does this architecture look correct @kuwv?

        "Architecture": "amd64",
        "Os": "linux",

The env also LGTM: "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/busybox", "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt" ],`

@spiffcs I set the entrypoint to [''] and put the absolute path for both /syft and /grype and was able to get it to work.

Is it possible to move these to some existing PATH?

Edit: Suggest maybe adding it to /usr/local/bin?

Hi @kuwv, sorry for the delay getting back to you here. We'll take and consider moving the binaries to the /usr/local/bin path. Maybe adding some symlinks from /usr/local/bin/ to the appropriate binaries in / would be a good solution.

👋 @kuwv check this repo out - it's a basic example on how to use gitlab with the debug image

https://gitlab.com/spiffcs/gitlab-ci-playground/-/blob/main/.gitlab-ci.yml?ref_type=heads

I followed this documentation here: https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#override-the-entrypoint-of-an-image

Job output for the above ci confiruration using /syft alpine:latest:

I can reopen this issue if you need more help, but just wanted to post the verification that our debug image works as intended with the provided distroless shell

anchore / syft

Unable to use Syft from GitLab CI/CD #1858