No timeout in dryRun()?

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Write a slow macro benchmark (e.g. large size, slow algorithm)
2. Run caliper
3. Caliper appears to freeze after "This selection yields %s experiments."

What is the expected output? What do you see instead?

I'd expect caliper to apply timeouts to the dryRun invocations, too; or give a 
chance to skip them.
Performing a dry run is cheap for microbenchmarks, but for macrobenchmarks its 
can be rather expensive. It would be nice, to be able to use the time limit -l 
to skip benchmarks that are too slow.

In my particular case, I'm benchmarking different sorting implementations out 
of curiosity. And of course I do have insertion sort and bubble sort in there, 
too; but these scale really bad. My overall goal is to estimate real (average) 
complexity for such implementations, so I wanted to have some measurements for 
large n in there, too, to see if that improves my curve fitting.

Original issue reported on code.google.com by erich.sc...@gmail.com on 7 Jul 2013 at 11:01

[GoogleCodeExporter]

So, I think there are a few things that I can pull out of this report:

1) dryRun has always been a simple invocation of the benchmark method.  Thus 
far we've been pretty happy with that.  Applying a timeout would at least 
require a different thread.  I'm not entirely opposed to that, but I'm not sure 
that skipping a "dry run" is really in the spirit of a "dry run".

2) Adding meaning to negative values of time limit seems odd to me.  If we were 
to put some sort of time limit specifically on dry runs, I don't think that 
we'd overload the time limit for that.

3) It sounds like there are times where you just want to run a subset of your 
parameters.  Dry runs seem orthogonal to which algorithms you select.  Can you 
just use the -D flag or the -b flag to select the correct experiments?

4) I think that adding messaging about dry runs specifically to the console 
output is a good idea.  At least then you would be aware that dry runs are 
happening.

Original comment by gak@google.com on 8 Jul 2013 at 5:45