[2d timebox] Stress test dotmesh

[alaric-dotmesh]

Our one stress-test of Dotmesh, dothub, routinely breaks messily. It would be nice if dotscience cloud doesn't have the same behaviour as we're planning on doing lots of demos with it soon, and even letting people have their own beta accounts on it.

Rather than waiting for trouble and scrambling around to fix it, let's jump the gun and have a proper stress test under controlled conditions, so we can find bugs and fix them!

Make it use Dotscience-like traffic patterns, but just go direct to Dotmesh, to keep the test simple and easy to recreate locally.

• Single-node cluster.
• Thousands of dots.
• Thousands of commits on those dots, with large commit metadata.
• Large files in dots (> 10G, see https://github.com/dotmesh-io/dotmesh/issues/655)
• Hundreds of different user accounts doing the RPCs
• Operations should be
  • pushes
  • pulls
  • S3 file uploads
  • S3 file lists
  • requests for the commit history of a dot DotmeshRPC.CommitsById
  • requests for the general details of a dot DotmeshRPC.Get
  • lists of dots in the system
  • S3 file downloads
  • Dot creation with initial commit all from non-admin accounts... anything else dotscience does a lot of?
• Have a number of client goroutines doing these, and logging the start and end of every Dotmesh RPC call to a single stream so we can easily see what operations were overlapping when something went nuts.
• The goroutines should check invariants as they go; error cases that stop the test would include unexpected errors from Dotmesh RPCs, unexpected results returned (zero commits when we know we've made commits, for instance), or a timeout on any RPC call.