automate integration tests

[nonsense]

Currently we are running smoke tests semi-manually on a given swarm version/commit, which involves:

1. build docker image and push docker image to public registry
2. make a deployment of swarm with a specific set of params:
   • persistence
   • store size
   • number of nodes in network
   • logs level
   • etc.
3. run smoke tests 1-2 min after nodes are up, so that we know subscriptions have been setup.
4. manually review results in grafana, kibana, kubectl and debug why they fail

---

It'd be nice to further automate this so that we get a shorter feedback loop.

It would also be beneficial to have a remote docker adapter, so that we run our simulation tests on a remote environment such as a k8s cluster.

What I am worried of is that if we automate this, I think that it is hard to know in advance what type of information we need to keep as part of 4, so that we debug a failed test run. Furthermore if we tear down a deployment, we also lose the ability to interactively inspect it.

Let's discuss how we can improve the current setup, because we all agree that there are too many manual steps in this process.

cc @holisticode @justelad @gluk256 @skylenet

[nonsense]

In an ideal world, I'd like to be able to have a job triggered on every push in Github, that runs a swarm private deployment of 50-100 nodes, runs the smoke tests on it for a specified period of time (or a simulation test?) (let's say between 1 run and 24*60 runs (overnight for releases)), and reports back success or fail as a Github status next to the commit. On a failed status, we should have the tools to be able to track what went wrong, which involves:

• recording topology and current connections
• recording and retaining stats (for at least 3-7 days)
• recording and retaining logs (for at least 3-7 days)
• recording and retaining traces (for at least 3-7 days)