Update Docker pipeline to build all dockerfiles

[denizgenc]

(Copied over from Trello)

Description

Our current Docker pipeline (https://github.com/alphagov/cyber-security-concourse-base-image/) only builds the base image at the moment. We should get it to build all the dockerfiles we need so as to have a sinlge place to keep all dockerfiles.

Checklist

• [x] http-api-resource
• [x] load-testing
• [x] csw
• [x] dhs
• [x] sec_adv
• [x] cyber_chalice
• [x] csls_concourse
• [x] concourse_worker_health
• [x] Fix pipeline triggering to check all jobs are triggered when the base image Dockerfile changes
  • This only affects csw-chalice, so this is done when #43 is fixed

Comments (from Trello)

pritchyspritch Mar 30 at 12:55 PM

Starting on Dan's magic http-api-resource container

pritchyspritch Apr 1 at 5:22 PM

Also added health monitoring for all tasks

pritchyspritch Apr 2 at 10:52 AM

DHS not in use by the looks of it

pritchyspritch Apr 2 at 11:56 AM

The CSW and Chalice ones are in here: https://github.com/alphagov/csw-concourse/tree/master/dockerfiles But they're blocked on TF11 upgrades. http-api-resource is blocked on me not knowing how to fix the concourse errors.

akinnane Apr 7 at 10:23 AM

Needs Dan's input.

[denizgenc]

In https://cd.gds-reliability.engineering/teams/cybersecurity-tools/pipelines/cyber-security-concourse-base-image I've noticed that the CSLS image is being built, but none of the others mentioned (CSW, Chalice, DHS, http-api-resource, and concourse worker health)

[denizgenc]

It's mentioned in the original issue post where the CSW and Chalice Dockerfiles can be found, but not the others.

Both http-api-resource and concourse-worker-health can be found in their respective directories here: https://github.com/alphagov/cyber-security-cloudwatch-config/tree/master/docker

Ignoring DHS as the original issue had a comment remarking that it was not being used.

[denizgenc]

Summary of the PR above as of today (2020-08-18):

• csw-concourse-worker, cyber-chalice and concourse-worker-health have been added
• http-api-resource errors in Concourse because the Dockerfile isn't defined as well as it could be

[danjoneslf]

I've moved this into "Up next" and added a task in the checklist to make sure the pipeline triggering is working correctly. Currrently the containers aren't all being rebuilt when the base image Dockerfile changes which means we have multiple different TF versions running in concourse.

[denizgenc]

The following PR allows the http-api-resource to be built in Concourse - though the problem now is that isort is throwing an error during the final step of the build, so there is still work to be done in getting that merged in https://github.com/alphagov/cyber-security-cloudwatch-config/pull/48

[denizgenc]

Fixed the build issues mentioned in the above comment with https://github.com/alphagov/cyber-security-cloudwatch-config/pull/50

[danjoneslf]

There's a remaining checkbox on the checklist - I think it may have been done already. Need to check.

[denizgenc]

The remaining item was "Fix pipeline triggering to check all jobs are triggered when the base image Dockerfile changes". Having a look through the related PR, it seemed that the only job that was affected was the one that built the csw-chalice image, which was split off into its own issue and PR (#41 and #43).

I think I'll make it so that #43 closes this issue as well.