Switch UKWA Docker image builds to standard workflow

[anjackson]

We need to make sure all important Docker images are scanned for security issues as part of the GitHub Actions process, before the images are pushed to Docker Hub.

To do this, we can reuse GitHub Actions workflows across repositories, to ensure we build, scan and upload Docker Images consistently.

This is an example of a container that uses the shared workflow: https://github.com/ukwa/ukwa-warc-server/blob/master/.github/workflows/push-to-docker-hub.yml

The task here is to go through the stacks in this repository and update every referenced container build to re-use this shared workflow. Every change should be proposed as a PR on each repository, and linked here for @anjackson to review.

[anjackson]

Note fragment is just

jobs:
  run_docker_build_workflow:
    uses: ukwa/ukwa-services/.github/workflows/push-to-docker-hub.yml@master
    secrets:
      DOCKER_HUB_USERNAME: ${{ secrets.DOCKER_HUB_USERNAME }}
      DOCKER_HUB_ACCESS_TOKEN: ${{ secrets.DOCKER_HUB_ACCESS_TOKEN }}

But the image name can be overridden if needed (see warc-server example)

[anjackson]

Switched to Trivy security scanner as this seems more widely used and a bit easier to work with.

[anjackson]

Note that this query can be use to check on which systems are using this shared workflow: https://github.com/search?q=org%3Aukwa+push-to-docker-hub&type=code

[ldbiz]

The following repos make up the w3act stack.

• backstage
• crawl-log-viewer
• docker-pdf2htmlex
• pywb
• solrwayback
• ukwa-notebook-apps
• ukwa-pywb
• ukwa-warc-server
• w3act

Of these, the following are third party services (which we contribute to), and have not been amended (this may change subject to guidance).

• solrwayback (upstream is netarchive)
• pywb (upstream is webrecorder)

The following are already using the workflow:

• ukwa-notebook-apps
• ukwa-pywb
• ukwa-warc-server

Repos with workflows updated by this work to the new standard are:

• backstage
• crawl-log-viewer
• docker-pdf2htmlex
• w3act
• ukwa-services (not in w3act stack but see note below)

Note:

1. w3act has some version tagging code that is now merged into ukwa-services as an optional parameter
2. crawl-log-viewer and docker-pdf2htmlex had no workflow before this change.
3. The work noted here covers the change to the shared workflow across the w3act stack. It does not include other repos, but will serve as a pilot.
4. It does not include improvements to the security process itself (eg. vulnerability scanning) which is being undertaken in parallel work.

[ldbiz]

I've started merging the PRs as its quite a bind to test in temporary branches.

[ldbiz]

All w3act repos now on the shared workflow within UKWA org main branches on github. Possibly a bit more testing to follow.

[ldbiz]

access stack repos in scope for workflow and if necessary changed in the above work (others in the stack had either already been done or were omitted after investigation:

crawl-streams ukwa-site ukwa-ui ukwa-ui-collections-dolr webrender-puppeteer

[ldbiz]

Miscellaneous repos that also needed converting to the new workflow and were done in the above work or omitted after investigation:

acid-crawl docker-airflow docker-hadoop docker-robot-framework ukwa-monitor ukwa-services (a new context parameter was required)

[ldbiz]

Testing note: Although there was significant testing early on, the later changes to ukwa-services to allow context passing have not been significantly regression tested. This can be done if necessary when we have decided on the priority services within the context of our renewed platform and processes.