nondeterministic order of printing results

[GoogleCodeExporter]

The order in which Caliper 0.5 prints results changes
nondeterministically between different runs of the same benchmark.
This makes comparing the results from different runs very difficult,
especially when there are many parameters, and each parameter has many
values.

I attached a test benchmark that exposes this behavior.  To get
different order of results, you need to run the benchmark many times
(e.g., 10-20 times).  If you run it only a few times (e.g., 2-3
times), it may happen that by accident the order is the same.

For the attached benchmark, for an execution the output is:
......................................................................
two one    ns linear runtime
  1   1 0.656 ==============================
  1   0 0.656 =============================
  0   1 0.656 =============================
  0   0 0.656 =============================
......................................................................

For another execution the output is:
......................................................................
one two    ns linear runtime
  1   1 0.656 =============================
  1   0 0.656 =============================
  0   1 0.657 ==============================
  0   0 0.656 =============================
......................................................................

In the first execution the order of parameters is "two one" while in
the second execution the order of parameters is "one two".  The
expected behavior is for the parameter order to be the same.

Original issue reported on code.google.com by adi1756...@gmail.com on 25 Sep 2012 at 6:50

Attachments:

• TestBenchmark.java

[GoogleCodeExporter]

The console output really isn't meant to be a good means for comparing output - 
the ordering certainly isn't the biggest limitation.  Viewing the results in 
the webapp (or accessing the raw JSON if you have your own analysis that you'd 
like to do) will certainly be better-suited for this task and both of those 
options have recently been improved for the soon-to-be-released v1.0.

Original comment by gak@google.com on 29 Oct 2012 at 5:35

• Added labels: Priority-Low, Type-Enhancement
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

Original comment by kevinb@google.com on 1 Nov 2012 at 8:32

[GoogleCodeExporter]

Original comment by gak@google.com on 1 Nov 2012 at 9:22

• Changed state: WontFix