Include all Dockerfiles in the project

[anskarl]

I am thinking that it may be practical to include as part of the project all Dockerfiles of images that are used in tests and additionally push all of them under a common namespace in Docker Hub (e.g., scruid/druid-docker, scruid/openresty and scruid/clusterf-chaos-proxy) or in GitHub Package Registry (beta) (tagged as docker.pkg.github.com/ing-bank/scruid/IMAGE_NAME:VERSION).

Dockerfiles will be part the project, will evolve with the project and publishing of the images can become a part of the CI process. Furthermore, when a new feature requires changes to some image (e.g., new version of Druid or some extension) the required change in the Dockerfile can be part of the corresponding PR.

A POC of such change can be found in https://github.com/anskarl/scruid/commit/1917a99df73d6bb7fbea525c0a408d5837093e38 in which:

• There are Dockerfiles for Openresty, Druid and Chaos-proxy, along with their configuration files
• A utility Makefile for building the required docker images (all together or individually)
• Uses GitHub Package Registry (beta)
• Just for demonstration purposes of this POC, all docker images are under docker.pkg.github.com/anskarl/scruid/ namespace (should change to docker.pkg.github.com/ing-bank/scruid/)
• For space efficiency all images are Alpine-based
• Images are highly configurable, for example can change the version of Druid
• When running ./services.sh start and some image is not available, it will try to build the image locally

What is missing for the POC:

• the preferred Docker registry (Docker Hub or GitHub Package Registry)
• the CI instructions to push the images to the registry (should require authentication)

What do you think? Do you find any benefits of making such an addition to the project?

[barend]

I think this is useful for the robustness of our build and control of our supply chain. On the downside, it makes it a bit more work to update our dependencies, but I think it's a net positive.

I would prefer to use the GitHub Package Registry because it's one fewer third-party.

[anskarl]

In https://github.com/anskarl/scruid/commit/7566fe6f2098b31859f57f9ceb22273bb143371e I've added GitHub Actions workflow for building the project similar to the current implementation that uses CircleCI, but the main difference is that it can also build Docker images for all the required external services (e.g., Druid) and pushed to GitHub Package Registry.

As it is also included in the commit message, the changes are outlined below:

• similar to the current CircleCI workflow:
  • workflow is triggered on push or pull request
  • .ivy2 directory is cached (hashed base on the checksum of build.sbt)
  • coverage report is being archived
• adds docker image publishing to github packages:
  • credentials are managed by github
  • no need to have separate user/space on Docker Hub
• use of common .env file for both docker-compose and Makefile. The versions of external services (e.g., Druid) are controlled here
• the workflow is portable, since forked repos will automatically publish images on their owner space
• simplifications on Makefile (no need for latest tag, automatically load the contents of .env file)

With GitHub Actions, the build flow can be handled without CircleCI and it is well integrated with GitHub Package Registry.

Please note that the images in docker-compose in https://github.com/anskarl/scruid/commit/7566fe6f2098b31859f57f9ceb22273bb143371e are pointing to GitHub Package Registry, therefore the automated tests in CircleCI are failing.

[krisgeus]

@anskarl this one can be closed right?

[anskarl]

Yes, I will close it.