Official read/write benchmark suite

[GoogleCodeExporter]

I suggest start building official read/write benchmark sute
so people can run the same set of tests and report back to your wiki;

http://groups.google.com/group/terrastore-discussions/browse_thread/thread/d2cf6
58e9dd01036

Original issue reported on code.google.com by Andrei.Pozolotin on 10 Mar 2011 at 3:04

[GoogleCodeExporter]

Agreed, candidates for the benchmark:

* Apache Benchmarking tool: http://httpd.apache.org/docs/2.3/programs/ab.html
* HttpPerf: http://www.hpl.hp.com/research/linux/httperf/
* JMeter: http://jakarta.apache.org/jmeter/

Any other options?

Original comment by sergio.b...@gmail.com on 12 Mar 2011 at 9:49

• Changed title: Official read/write benchmark suite
• Changed state: Accepted
• Added labels: Type-Feature, Priority-Medium, Project-Terrastore

[GoogleCodeExporter]

1) I prefer to stay with java and hence vote for jmeter :-)
(you can even embed it in the tool suggested below)

2) people frequently get shy when you ask them to run full blown tools like 
jmeter and can easily mess up results just by "inapropriate run";
hence jmeter test plans should come as part of terrastore build

3) I also suggest to include in terrastore build minimal command line tool that 
will be published to maven central, will produce simple csv result file which 
can be postted on terrastore wiki / google docs for comarison;

4) also, this "terra tool" could rely on sigar or similr:
http://www.hyperic.com/products/sigar 
to collect hardware info so results from different people can be comared in 
absolute values.

5) finally you may provide some terra bucket somewhere, so the results of these 
runs are dumped there automatically, like it is done here: 
http://global.phoronix-test-suite.com/

Original comment by Andrei.Pozolotin on 14 Mar 2011 at 8:05

[GoogleCodeExporter]

Andrei, those are really good points: we'll try to work on them and see what we 
come up with.

Original comment by sergio.b...@gmail.com on 15 Mar 2011 at 5:28

[GoogleCodeExporter]

We could start by:

1) Creating a JMeter test file for write, read and mixed read/write scenarios.
2) Creating a shell script to run the test.
3) Providing a way to simply plot JMeter results.

Comments?

Original comment by sergio.b...@gmail.com on 15 Mar 2011 at 5:31

[GoogleCodeExporter]

As an alternative, we could integrate Terrastore tests into a third party 
benchmarking suite such as:

1) Radargun: http://sourceforge.net/apps/trac/radargun/wiki/WikiStart
2) YCSB: https://github.com/brianfrankcooper/YCSB/wiki/

Comments?

Original comment by sergio.b...@gmail.com on 15 Mar 2011 at 5:34

[GoogleCodeExporter]

re: #4 - cool

re: #4.3 - there are some plugins for jmeter: 
http://rubenlaguna.com/wp/better-jmeter-graphs/

Original comment by Andrei.Pozolotin on 15 Mar 2011 at 8:39

[GoogleCodeExporter]

re: #5

this:
https://github.com/brianfrankcooper/YCSB/wiki/Papers-and-Presentations
http://research.yahoo.com/node/3202
http://research.yahoo.com/files/ycsb.pdf

makes ycsb look very credible and relevant to terrastore eco space;
but ycsb probably is hard for "mere mortals"

here is a compromise:

1) ask Brian Cooper to include terrastore into his load test setup in yahoo labs
and ask him give you permissions to publish his benchmarks on your site;
too bad you will have to compete agains big table, etc :-)

2) provide jmeter for the rest of us :-)

Original comment by Andrei.Pozolotin on 15 Mar 2011 at 8:51

[GoogleCodeExporter]

Regarding JMeter, I've just discovered this one from your link: 
http://code.google.com/p/jmeter-plugins/
That makes JMeter a strong candidate :)

Regarding YCSB, you're right it may be hard to setup.

So, let's start with JMeter and see what we come up with.

In the next few days, I will outline a first JMeter test plan and share it with 
you here, so to gather some feedback: is that okay for you?

Original comment by sergio.b...@gmail.com on 16 Mar 2011 at 1:23

[GoogleCodeExporter]

sounds great, thank you.

Original comment by Andrei.Pozolotin on 16 Mar 2011 at 3:42