search

zeebe-io / zeebe-chaos

Contains everything related to chaos engineering in Zeebe, which means chaos experiments, hypothesis backlog etc.
https://zeebe-io.github.io/zeebe-chaos/
Apache License 2.0
23 stars 4 forks source link

Define how to test go fault injector #140

Closed Zelldon closed 2 years ago

Zelldon commented 2 years ago

See example code in #122

We should define a way/strategy to test the provided functionality in an automated way via github actions

related to #139

Zelldon commented 2 years ago

Test Strategy

Before we can define the test strategy we need to know which functionalities we want to build and support, so let's shortly summarize that.

Features

Current state

We have written a lot of bash scripts that use kubectl to interact with a Kubernetes cluster and zbctl to interact with a deployed Zeebe cluster.

We have scripts to cause failures, like restarting the leader or causing CPU pressure on a node. Furthermore, we have scripts to create new process instances, deploy processes, etc. on a Zeebe Cluster.

The scripts are called by chaosToolkit or by a kotlin worker, to execute chaos experiments. We want to migrate everything into one application / project, see https://github.com/zeebe-io/zeebe-chaos/issues/121#issue-1161059002

Summary

This means:

Testing

With the understanding of what we need to support we can think about testing. Important is the part that it needs to communicate with some external systems.

Unit Tests:

CLI

Ideally, we could test the CLI separately from the backend and just verify whether the parameters are correctly forwarded.

Maybe this https://gianarb.it/blog/golang-mockmania-cli-command-with-cobra is helpful

Job Worker

Would be interesting whether we can test this with Testcontainers and the zeebe-process-test or eze. Here we want to potentially also verify our processes and whether the job workers are correctly called and call the backend correctly.

:thinking: Maybe we also think about it differently and implement the job handlers and test the handlers only via giving different inputs (jobs)

Backend - Mocking

One way would be to mock the systems away and validate that the expected requests are sent. For Zeebe this, I think, is quite simple (we do the same in the go/java/c#) tests, where we just start a gRPC server and send responses back.

Maybe we could also just open a socket and receive the input and validate that with some golden files :thinking:

:thinking: It might make sense to abstract the k8 access away, in such a way that we just validate whether we give the correct values to the API. Because we can expect that the K8 API works and is tested. Similar to Zeebe.

Integration

To not access our GKE cluster and set up a complete Zeebe cluster etc., which would take quite a while, we could try it with KIND and starting here a small zeebe cluster. Note: I would expect that we build this on a later stage and start with the smaller unit tests.

:thinking: Since it would be used by several other applications this would be another step of the integration test (e.g. Testbench).

deepthidevaki commented 2 years ago

  • We want to build a CLI, which can be run locally + and via chaosToolkit

  • We want to have a job worker, which can be used with testbench and e2e tests

  • Ideally we have a backend library that is used by the components above

  • The backend needs to support the communication with a Kubernetes cluster and a Zeebe cluste

:+1:

Here we expects the testbench and e2etest to use the same job workers, right? Meaning that e2e-test doesn't need to implement workers using the re-usable library.

Zelldon commented 2 years ago

Yes exactly, similar to how it is right now.

deepthidevaki commented 2 years ago

Job Worker

Would be interesting whether we can test this with Testcontainers and the zeebe-process-test or eze. Here we want to potentially also verify our processes and whether the job workers are correctly called and call the backend correctly.

thinking Maybe we also think about it differently and implement the job handlers and test the handlers only via giving different inputs (jobs)

If we test the re-usable library and the job workers are just a wrapper around them, do we have to test the workers extensively. In my mind the job-workers wouldn't do much - but just execute the library method and complete the job. So there wouldn't much logic in them.

Zelldon commented 2 years ago

Agree it would be just more like: does it call the right things.

Zelldon commented 2 years ago

This is done and first go ci was added and first tests with #122