Automatically test the whole system

We do have some unit tests and integration tests for parts of shutter. I'd like to be able to automatically test the whole system.

I've taken a look at jepsen, which is framework for testing distributed databases. It's being used to test failure scenarios like network partitions, clock skew and dying processes. It's not a perfect match since what we're testing is not exactly a distributed database, at least not in the sense that jepsen expects. Nevertheless I would expect us to be able to reuse some of the code it provides. One interesting fact is that jepsen has been used to test tendermint.

I've watched some talks on distributed system testing and simulation testing, e.g. Testing Distributed Systems w/ Deterministic Simulation and Simulation Testing.

The first talk is about how foundation db is being tested. They run their whole system inside a single process. They can reproduce failures easily because everything is run in a deterministic way besides seeding the random number generator. They do instrument their code and have written their own language for running the tests. Unfortunately this is something that seems impossible for us to implement in a timely manner - if at all.

The second talk is about a solution using the clojure simulant library. What they do have in common with jepsen is that they have a separate steps for generating test cases deterministically, running those test cases and reporting results.

Contrary to what we do in the unit tests, I think we should let go of having each test test exactly one thing. It's rather run the system until some final state is reached and then look for multiple invariants that may be broken.

My current plan is to implement a clojure library, which starts the whole system locally, interacts with it and examines the state via querying each programs postgresql database, i.e. this will not be black box testing. Instead we take advantage of having the data stored in an easily accessible database. We may even change our programs to log interesting/important events to the database.

Using clojure has the possible benefit that we may reuse some of jepsen's code as well as our own code from the play directory.
Implementing it in go would be possible, but I prefer a higher level language for those tests. Using python is nothing a sane person would ever do for various reasons, so I don't consider that a viable option.

@ulope @cducrest @jannikluhn Comments or objections on the choice of language or the rest of my plan?

TODO: ~~- [ ] Implement a go program that watches the contracts for events and imports them into a database~~ I was able to just use the keypers database for that in one case. maybe that's already good enough

[x] Write a setup function that sets up a system with only the keypers and ganache
[x] test issue #184
[ ] test with decryptors and the mock node running

shutter-network / rolling-shutter

Automatically test the whole system #186