Longevity/stress tests for brooklyn

[aledsage]

We need more longevity and stress tests in Brooklyn.

Example test As a first test to write, how about:

• create a web-cluster (nginx + as7)
• a resize policy to scale-out and scale-back the cluster (from 1 to 4 app-servers)
• an "enricher" to publish a pseudo-load sensor that varies sinusoidally over time so that the resizing will be triggered every couple of minutes.
• a scheduled executor for sending http requests, to confirm the web-app is always accessible via the nginx node

We'd programmatically monitor:

• scale up/down keep happening (ps aux | grep jboss returns different pids over time)
• memory usage of brooklyn (jmap -histo )
• grep -E "SEVERE|ERROR|WARN|Exception|Error" brooklyn.log We should strive for that to be empty!

If watching manually, we'd also run jconsole.

We'd first run this on localhost. Then we'd run it with a fixed set of IPs if we have access to a few spare servers.

Test structure As a first step, we can write a script that launches both brooklyn and a "monitoring process" to do the above checks. Groovy is probably a good bet for the monitoring process code, because it is scripting.

As a future step, we would beg a server (or few) for running this regularly. We'd have jenkins run a testng test for it (that called the scripts?). Jenkins would run it every day, and the test would run for 23 hours.

Improving test coverage There are various other things I'd love to be covered in the longevity tests. These include:

• brooklyn web-console
• more entity types
• more policies / enrichers
• entities that query sensor-values using jmx, http, and scripts

[aledsage]

On the brooklyn-dev mailing list, Alex posted the following:

Looks great!

Maybe first do a starter test which creates a simpler app (just an as7, and one with just a tomcat for jmx testing) and soaks it -- to make no leaks in collecting sensors.

Optional extension with pings the web console to make sure no leaks there.

If fixture (and web-check extension) are easily reusable that's a big plus of course!