Automated Load Testing and Failure Testing

[johnsonj]

• Verify QPS
• Verify dropped messages are handled gracefully
• Can we simulate cascading failure?

[evandbrown]

• Verify QPS

We need to get an idea of a baseline QPS. We can start by setting parameters on what the test environment looks like, tweak params as needed, and eventually get regression testing out of this. Here's a stab at the test environment:

• Every VM type discussed below is n1-standard-2
• A CF environment where each service runs on separate VMs,
  • 5 Diego Cells
  • 1 Loggregator instance
• A stackdriver-tool nozzle deployed
  • 1 instance
  • Configured to receive and push all Firehose metrics to Stackdriver

Parameters should be adjusted to find the upper bound of QPS where some percentage (99?) of queries succeed.

• Verify dropped messages are handled gracefully

After an upper bound QPS is established, the 2nd phase of a load testing pipeline should push past that bound, triggering failure in the form of dropped messages by the nozzle. Graceful handling of dropped messages is characterized by the nozzle logging the fact that it dropped messages via the fluentd client.

*Can we simulate cascading failure?

Deploy 2 nozzles to handle ~2x QPS, and scale the CF install to provide that QPS. Resize the nozzle job to only have 1 VM and observe its behavior. It should log what it's able to log and report dropped messages without crashing.

@johnsonj how's the for an initial outline?

[johnsonj]

LGTM

[johnsonj]

Closing due to age. This would be nice to have.