add a tree-like system to run sets of benchmarks

[c-cube]

A system a bit similar to OUnit, with the ability to select only a subset of the benchmarks to run them. As each benchmark can take a significant time, it is convenient to be able to run only the ones that are relevant to some function, say, if we are trying to improve this function's performance.

There is also a small patch to _oasis because it wouldn't compile with a recent version of oasis otherwise.

[Chris00]

The interface has become something I like. I still want to read the implementation a bit more closely before merging. Thank you for your patience!

[c-cube]

Glad to hear it! :)

[c-cube]

I am tired of this. Do you have a better idea of how to represent paths to benchmarks?

[Chris00]

On Mon, 1 Dec 2014 05:33:54 -0800, Simon Cruanes wrote:

I am tired of this. Do you have a better idea of how to represent paths to benchmarks?

Sorry, I have to postpone this still for a little while — I have more urgent things to imperatively finish first.

[Chris00]

I merged your patch with some modifications in the tree banch. Let me know whether this fits your needs.

[c-cube]

I've tried adapting my microbenchmarks to this branch, the result is here. You might want to take a look to see how I use the various combinators. The point is that this makes a lot of benchmarks and running them all when tuning one particular algorithm is too slow.

A rough edge that remains: with_int is definitely useful to me, but right now I either have to adapt the path to the number by hand (I made it at the beginning, see here for instance), or to ignore the number.

Also, the way the tree is printed is nice, but has some strange-looking newlines:

 % ./run_benchs.native --tree              
No benchmark at root
- batch

  - array

    - batch: 3 benchmarks
  - klist

    - batch: 3 benchmarks
  - list

    - batch: 3 benchmarks
- cache

  - fib: 5 benchmarks
- iter

  - flat_map: 2 benchmarks
  - fold: 4 benchmarks
  - iter: 2 benchmarks
- list

  - append

    - append_100: 1 benchmark
    - append_10000: 1 benchmark
    - append_100000: 1 benchmark
  - flat_map

    - flat_map_100: 1 benchmark
    - flat_map_10000: 2 benchmarks
  - flatten

    - flatten_100: 1 benchmark
    - flatten_10000: 1 benchmark
    - flatten_100000: 1 benchmark
- tbl

  - add: 4 benchmarks
  - find: 5 benchmarks
  - replace: 4 benchmarks
- vector

  - append: 3 benchmarks
  - map: 3 benchmarks

[Chris00]

On Mon, 22 Dec 2014 14:27:39 -0800, Simon Cruanes wrote:

I've tried adapting my microbenchmarks to this branch, the result i s here. You might want to take a look to see how I use the various
combinators. The point is that this makes a lot of benchmarks and r unning them all when tuning one particular algorithm is too slow.

I understand the point. I'll try to take a look tomorrow.

A rough edge that remains: with_int is definitely useful to me, but right now I either have to adapt the path to the number by hand (I made it at the beginning, see here for instance), or to ignore the number.

There was no semantics explained in the mli so I chose to ignore the
number. I can prefix the tree “f i” by the path numbered “i” it you
want (the trees “f i” can possibly have many different paths, I do no t want to require that they are all the same).

I was also thinking that "" in the middle (or end) of a path could be treated as a wildcard. That means that ["a"; "b"] would only run th e benchmarks at that level, one would need to specify ["a"; "b"; ""]
to run the subtree. What du you think?

Also, the way the tree is printed is nice, but has some strange-loo king newlines:

This was due to a typo. I have pushed a fix. Please try again.

[c-cube]

Much better. I like it overall. I agree that the semantics of with_int w.r.t. paths is still a bit unclear :/

About "" as a wildcard, does that mean that I have to write --path foo.bar. instead of --path foo.bar to run the whole subtree?

[Chris00]

With the latest version of benchmark, the output of ./run_benchs.native --tree is

No benchmark at root
- batch
  - array
    - batch: 3 benchmarks
  - klist
    - batch: 3 benchmarks
  - list
    - batch: 3 benchmarks
- cache
  - 10
    - fib: 1 benchmark
  - 100
    - fib: 1 benchmark
  - 1000
    - fib: 1 benchmark
  - 20
    - fib: 1 benchmark
  - 200
    - fib: 1 benchmark
- iter
  - 100
    - fold: 1 benchmark
  - 1000
    - flat_map: 1 benchmark
    - fold: 1 benchmark
    - iter: 1 benchmark
  - 10000
    - flat_map: 1 benchmark
    - fold: 1 benchmark
    - iter: 1 benchmark
  - 1000000
    - fold: 1 benchmark
- list
  - append
    - 100
      - append_100: 1 benchmark
    - 10000
      - append_10000: 1 benchmark
    - 100000
      - append_100000: 1 benchmark
  - flat_map
    - 100
      - flat_map_100: 1 benchmark
    - 10000
      - flat_map_10000: 2 benchmarks
  - flatten
    - 100
      - flatten_100: 1 benchmark
    - 10000
      - flatten_10000: 1 benchmark
    - 100000
      - flatten_100000: 1 benchmark
- tbl
  - 10
    - add: 1 benchmark
    - find: 1 benchmark
    - replace: 1 benchmark
  - 100
    - add: 1 benchmark
    - find: 1 benchmark
    - replace: 1 benchmark
  - 1000
    - add: 1 benchmark
    - find: 1 benchmark
    - replace: 1 benchmark
  - 10000
    - add: 1 benchmark
    - find: 1 benchmark
    - replace: 1 benchmark
  - 20
    - find: 1 benchmark
- vector
  - 100
    - append: 1 benchmark
    - map: 1 benchmark
  - 10000
    - append: 1 benchmark
    - map: 1 benchmark
  - 100000
    - map: 1 benchmark
  - 50000
    - append: 1 benchmark

[c-cube]

That's neat. It requires a kind of wildcard though: to run all benchmarks for, say, tbl.add, it should either ignore the numbers (so --path tbl.add works) or provide a "" branch so that `--path tbl..add` works.

[Chris00]

Since benchmarks without names now have a clear semantic, I wonder whether it would not be better to use, say !@, to lift a samples Lazy.t to a Tree.t and use @> and @>> to give names... Now "name" @> !@ bench looks a bit funny...

[Chris00]

I changed a bit the run function — the one in place was not versatile enough I thought. (In the process, I also allow repeated -p options on the command line.)

[Chris00]

I think I am about ready to merge unless you have further remarks.

[Chris00]

Since I have to check path names in order to ensure they do not contain '.', maybe it is better to allow "a.b.c" and split them. I think this is convenient.

[c-cube]

Thanks for your work on integrating this! The interface is more polished now. I'll be away from any computer during the holidays, but I think it's ready to merge. I hope this will be useful. Merry Christmas to you!

[Chris00]

I have implemented the “names containing dots being interpreted as paths” and the the “wildcard patterns”. I'll likely merge later tonight. Merry Christmas to you too!