search

ukwa / ukwa-services

Deployment configuration for all UKWA services stacks.
Apache License 2.0
5 stars 5 forks source link

Switch UKWA Docker image builds to standard workflow #77

Closed anjackson closed 7 months ago

anjackson commented 2 years ago

We need to make sure all important Docker images are scanned for security issues as part of the GitHub Actions process, before the images are pushed to Docker Hub.

To do this, we can reuse GitHub Actions workflows across repositories, to ensure we build, scan and upload Docker Images consistently.

This is an example of a container that uses the shared workflow: https://github.com/ukwa/ukwa-warc-server/blob/master/.github/workflows/push-to-docker-hub.yml

The task here is to go through the stacks in this repository and update every referenced container build to re-use this shared workflow. Every change should be proposed as a PR on each repository, and linked here for @anjackson to review.

anjackson commented 2 years ago

Note fragment is just

jobs:
  run_docker_build_workflow:
    uses: ukwa/ukwa-services/.github/workflows/push-to-docker-hub.yml@master
    secrets:
      DOCKER_HUB_USERNAME: ${{ secrets.DOCKER_HUB_USERNAME }}
      DOCKER_HUB_ACCESS_TOKEN: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

But the image name can be overridden if needed (see warc-server example)

anjackson commented 1 year ago

Switched to Trivy security scanner as this seems more widely used and a bit easier to work with.

anjackson commented 1 year ago

Note that this query can be use to check on which systems are using this shared workflow: https://github.com/search?q=org%3Aukwa+push-to-docker-hub&type=code

ldbiz commented 10 months ago

The following repos make up the w3act stack.

Of these, the following are third party services (which we contribute to), and have not been amended (this may change subject to guidance).

The following are already using the workflow:

Repos with workflows updated by this work to the new standard are:

Note:

  1. w3act has some version tagging code that is now merged into ukwa-services as an optional parameter
  2. crawl-log-viewer and docker-pdf2htmlex had no workflow before this change.
  3. The work noted here covers the change to the shared workflow across the w3act stack. It does not include other repos, but will serve as a pilot.
  4. It does not include improvements to the security process itself (eg. vulnerability scanning) which is being undertaken in parallel work.
ldbiz commented 10 months ago

I've started merging the PRs as its quite a bind to test in temporary branches.

ldbiz commented 9 months ago

All w3act repos now on the shared workflow within UKWA org main branches on github. Possibly a bit more testing to follow.

ldbiz commented 7 months ago

access stack repos in scope for workflow and if necessary changed in the above work (others in the stack had either already been done or were omitted after investigation:

crawl-streams ukwa-site ukwa-ui ukwa-ui-collections-dolr webrender-puppeteer

ldbiz commented 7 months ago

Miscellaneous repos that also needed converting to the new workflow and were done in the above work or omitted after investigation:

acid-crawl docker-airflow docker-hadoop docker-robot-framework ukwa-monitor ukwa-services (a new context parameter was required)

ldbiz commented 7 months ago

Testing note: Although there was significant testing early on, the later changes to ukwa-services to allow context passing have not been significantly regression tested. This can be done if necessary when we have decided on the priority services within the context of our renewed platform and processes.