Memory performance testing rig

[jwreagor]

We (I) need to learn from both #464 and #521 by incorporating the knowledge gained from both issues into a sub section of the project.

I'd like to automate as much performance monitoring, as possible, into an easy to construct testing rig that we can use to preemptively monitor and forecast the memory health of the application. The goal is to script an ephemeral machine to stand-up and soak test prior to major releases. Reporting should be stored for reference and kept for comparison against historical test runs.

The testing rig work should include the following...

• A special build time argument that enables net/http/pprof and/or possibly gops at run time. Use the telemetry endpoint for harvesting internal ContainerPilot stats.
• Terraform configuration to build out a native Linux KVM instance to run Docker under.
• A Docker container that provides all tools and scripts required by various tests.
• A few common cases of running ContainerPilot including a pseudo-normal app, isolated config features, reload and signal handling tests.
• The ability to gather heap profiles over an extended period of time.
• Graphs that make it easy to distinguish progressive increases in memory usage.
• Historical references backed up onto Manta for future comparison and inspection.

[jwreagor]

Ignoring a lot of wishful thinking above... I created a repository that includes a testing environment one can use to exercise ContainerPilot and harvest profiles for further analysis. I think that'll sum this up for the moment.