Add Trivy vuln scanning to Concourse docker containers

[denizgenc]

[Description and comments are copied over from the Trello ticket found here: https://trello.com/c/EVzy1zE7/263-add-trivy-vuln-scanning-to-concourse-docker-containers ]

The plan is to create a pipeline that runs very regularly - e.g. hourly. It will have Trivy scan for vulnerabilities in the "one true" docker container that Tris and Alice are currently working on (see https://trello.com/c/Aq8qX4af/193-create-a-shared-docker-image-for-concourse-pipelines).

If no vulnerabilities are found, then no action is taken. If vulnerabilities are found, it triggers a rebuild of the docker container (which would otherwise be built every week or so).

(Trivy can be found at: https://github.com/aquasecurity/trivy)

[denizgenc]

@gds-ahine at 2020-04-02T11:38

---

Looks like Deniz has done a lot of comprehensive legwork on this already.

From initially playing with this I can concur that our base concourse image, even after adding backports and doing a dist-upgrade etc is at the moment returning 4 critical vulnerabilities for linux-libc-dev package.

  root@36bf71f73d42:/tmp# apt-get update
  Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
  Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
  Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
  Hit:4 http://archive.ubuntu.com/ubuntu trusty-backports InRelease
  Hit:5 http://security.ubuntu.com/ubuntu bionic-security InRelease
  Reading package lists... Done
  root@36bf71f73d42:/tmp# apt-get upgrade
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  Calculating upgrade... Done
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
  root@36bf71f73d42:/tmp# apt-get dist-upgrade
  Reading package lists... Done
  Building dependency tree
  Reading state information... Done
  Calculating upgrade... Done
  0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

Guess we need to triage these vulnerabilities, determine the risk etc

Will do some more digging, presume the --ignore-unfixed will ignore linux-libc-dev criticals for our image, but any packages patched in the future for Ubuntu 18.04 (bionic) that had a CVE severity of critical would be flagged up.

From help -:

--ignore-unfixed display only fixed vulnerabilities [$TRIVY_IGNORE_UNFIXED]

Testing our base concourse image on my machine -:

trivy --light --exit-code 1 --severity CRITICAL gdscyber/ah-test-image:1.2

   gdscyber/ah-test-image:1.2 (ubuntu 18.04)
   =========================================
  Total: 4 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 4)

  +----------------+------------------+----------+-------------------+--- 
  ------------+
   |    LIBRARY     | VULNERABILITY ID | SEVERITY | INSTALLED VERSION | 
   FIXED VERSION |
  +----------------+------------------+----------+-------------------+--- 
  ------------+
  | linux-libc-dev | CVE-2019-10220   | CRITICAL | 4.15.0-91.92      |               
  |
  +                +------------------+          +                   +---------------+
  |                | CVE-2019-19813   |          |                   |               |
  +                +------------------+          +                   +---------------+
  |                | CVE-2019-19814   |          |                   |               |
  +                +------------------+          +                   +---------------+
  |                | CVE-2019-19816   |          |                   |               |
  +----------------+------------------+----------+-------------------+--- 
 ------------+

trivy --light --ignore-unfixed --exit-code 1 --severity CRITICAL gdscyber/ah-test-image:1.2

   gdscyber/ah-test-image:1.2 (ubuntu 18.04)
   =========================================
   Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)

[denizgenc]

@gds-ahine at 2020-04-02T12:05

---

Although we don't currently have private repos in Docker Hub and the pipeline already has the credentials to push up new images, not sure how critical it is to get the Trivy Auth functionality working.

Present, our Docker Hub account only permits a max of 1 private repos.

Will attempt to test this in any case.

[denizgenc]

@gds-ahine at 2020-04-02T16:57

---

Tested Trivy Authentication by making gdscyber/ah-test-image repo private.

Initial test was to run the following in the pipeline in the shell (job #195)

trivy --light --quiet --ignore-unfixed --exit-code 1 --severity CRITICAL gdscyber/ah-test-image:1.2

Didn't error or return anything, which is kind of expected.

Added the following parameters to the pipeline, re-ran and it was able to scan the image (ref job #196)

   params:
      TRIVY_AUTH_URL: https://registry.hub.docker.com
      TRIVY_PASSWORD: ((dockerhub_password))
      TRIVY_USERNAME: ((dockerhub_username))

Still need to test with pushing up an updated image. Figure out how we can test patching a vulnerable image.

Updated pipeline and pushed up to the trivy-scanning branch -:

https://github.com/alphagov/cyber-security-concourse-base-image/commit/fa808f859003703718a866f7d5e2bc662f802d50

[denizgenc]

@gds-ahine at 2020-04-03T16:42

---

Not much progress on this today, I've added a commit to the trivy-scanning branch.

https://cd.gds-reliability.engineering/teams/cybersecurity-tools/pipelines/trivy_scanning

Test pipeline fortrivy-scanning\build_and_scan_docker_image_for_vulnerabilities builds a Docker image from a test Dockerfile (our base image) and then scans with trivy. Tested with both --ignore-unfixed included and without including this flag.

To test pushing the image, --ignore-unfixed needed to be in the config.

I'm not sure how we really triage vulnerabilities with this, agree with Deniz about having reservations using the --ignore-unfixed flag. At present if we don't use this flag it detects the following CVE's for linux-libc-dev classified as CRITICAL vulnerabilities for our base Concourse image -:

https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10220 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19813 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19814 https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-19816

If we use --ignore-unfixed flag, it will not detect these vulnerabilities, will it detect future vulnerabilities in Ubuntu 18.04 of ?

https://cd.gds-reliability.engineering/teams/cybersecurity-tools/pipelines/cyber-security-concourse-base-image

The pipeline above, which builds the base image and other images doesn't automatically run each day, maybe we add a timer to this to build once a day, couple of times a day or alternatively if we're able to somehow check if for example the ubuntu:18.04 Docker image has been recently updated ?

https://github.com/concourse/docker-image-resource/issues/217