Proposals for CI platform

[Zarquan]

We need a Continuous Integration system that runs automated tests each time we create a new deployment. This is the first step along that route, to propose solutions and describe how they would work.

Given we are putting our code into GitHub the obvious one is the GitHub Actions CI system.

• https://help.github.com/en/actions/automating-your-workflow-with-github-actions/about-continuous-integration
• https://github.blog/2019-08-08-github-actions-now-supports-ci-cd/
• https://techcrunch.com/2019/08/08/github-actions-is-now-a-ci-cd-service/

but there is a whole range of others to choose from as well

• https://github.com/marketplace/category/continuous-integration

Question is .. how do we integrate the CI system with a deployment on an external OpenStack system that isn't part of the GitHub universe ?

[stvoutsin]

I've had a go creating a CI system for the project here: https://github.com/stvoutsin/zeppelinTo

Notes on how to setup a workflow can be seen here:https://github.com/stvoutsin/aglais/commit/11e42b384017006ac29270ce54a2ddac9e5df0f6

And results of the unit tests which are run on each commit can be found here: https://github.com/stvoutsin/zeppelinTo/actions

Setting up a CI system for deploying a Spark/K8s cluster will be a bit more complicated obviously, will have to think about what are unittests are, how we deploy the cluster in an automated way (Ansible, Dockerfiles etc..) and how we trigger the unittests to run something like Spark jobs on the cluster we create.

[Zarquan]

Setting up a CI system for deploying a Spark/K8s cluster will be a bit more complicated obviously, will have to think about what are unittests are, how we deploy the cluster in an automated way (Ansible, Dockerfiles etc..) and how we trigger the unittests to run something like Spark jobs on the cluster we create.

Tools like ZeppelinTo will be a useful component in the test infrastructure.

If all of the components, OpenStack, Magnum, Kubernetes, HDFS, YARN, MapReduce, Spark and Zeppelin either have command line clients or REST endpoints, then we can gradually work towards an automated test system by adding tests to each layer as we build it..

It might be that the best way of developing the system is as a set of smaller components developed as separate GitHub ~projects~ repositories, each of which has it's own test framework and unit tests. We then put them together to deploy a full system, which then another set of integration tests. See #34

[Zarquan]

We can link a self hosted test runner to the GitHub triggers, but there are security issues with doing that. https://help.github.com/en/actions/hosting-your-own-runners/adding-self-hosted-runners