Wind Tunnel scenario execution

[ThetaSinner]

Wind Tunnel is most useful when scenarios are run regularly in the same environment. There is already a Nomad cluster which has been set up for testing and can be used for this purpose.

Completion of this issue requires:

• [ ] A scenario can be configured as code to be deployed and run on a given number of nodes.
  • Note: The code deployed is monomorphic, but can behave differently depending on included configuration.
• [ ] The required infrastructure for reporting results is deployed (InfluxDB + Telegraf).
  • Note: We may want a "run-id" to be included with the reporting, so we don't have to clear these between runs and it is still easy to identify which metrics / logs go with each scenario run.
• [ ] Target nodes can be cleaned after a run. E.g. so that Holochain is stopped and its data cleared between runs.
• [ ] The end of a scenario is understood by the control plane, so that scenarios can be left to run back-to-back without needing manual intervention.
• [ ] There are notifications produced when something goes wrong with a scenario or something prevents running the full set of scenarios.

[ThetaSinner]

Best estimate for this issue is 2 weeks.

[neonphog]

We might need to spike/design more to understand how to implement these requirements: there is currently some flux with tfgrid development, etc.

[steveej]

we should be able to develop scenarios without requiring access to a big real-world execution cluster.