Proposed revamp of command-line interface

[GoogleCodeExporter]

I propose to clean up the command-line interface. The help text would look like 
this:

 -h, --help        print this message
 -n, --dry-run     instead of measuring, execute a single 'rep' of each
                   benchmark scenario in-process (for debugging); only options
                   -m, -d and -D are available in this mode
 -i, --instrument  comma-separated list of measuring instruments to use;
                   options include 'time', 'allocation', and 'memory_size'
 -w, --warmup      minimum time in seconds to warm up before each measurement;
                   a positive integer (default 10)
 -m, --method      comma-separated list of benchmarks to run; 'foo' is an
                   alias for 'timeFoo' (default: all found in class)
 -v, --verbose     generate extremely detailed logs and include them in the
                   result datafile (does not change console output)
 -t, --trials      number of independent measurements to take per benchmark
                   scenario; a positive integer (default 1)
 -b, --jre-base    base directory that JRE homes specified with --jre are
                   considered relative to
 -j, --jre-home    comma-separated list of JRE home directories to test on;
                   each may be either absolute or relative to --jre_base
                   (default: only the JRE caliper itself is running in)
 -o, --output      name of file or directory in which to store the results
                   data file; if a directory a unique filename is chosen; if a
                   file it is overwritten
 -s, --score       also calculate and display an aggregate score for this run;
                   higher is better; this score can be compared to other runs
                   with the exact same arguments but otherwise means nothing
 -d, --delimiter   separator used to parse --jre, --measure, --benchmark, -D
                   and -J options (default: ',')

... with -D and -J the same.  Changes:

* Short forms for every option
* --booleanOption=(true/false) supported, so command-line args can override 
default args from a file
* Hyphenated GNU-style (jre-base, not jre_base or jreBase)
* --debug becomes --dry-run
* -Dbenchmark= becomes dedicated parameter --method
* --captureVmLog becomes --verbose
* --warmupMillis becomes --warmup and is in seconds
* --vm becomes --jre-home + --jre-base, and can't contain jvm args
* --saveResults becomes -o/--output (just seemed more standard... open to 
feedback)
* --measureMemory/--measurementType/--primaryMeasurementType become just 
--instrument (the first one listed is treated as primary)
* --timeUnit, --memoryUnit : do we need these or can they be caliperrc-only? 
they don't affect what the tool does, just a bit about its console display
* --instanceUnit : I don't know that we need this at all.
* --runMillis : do we really need this?  what are we really trying to support 
here?
* --marker : this isn't supposed to be caliper option at all as far as I can 
tell, but is particular to communication to the worker process.
* --debug-reps : do we really need this?  
* --uploadResults : getting rid of this one for sure; it should be a separate 
program

Original issue reported on code.google.com by kevinb@google.com on 10 Feb 2011 at 7:32

[GoogleCodeExporter]

Excellent, Nice improvement.

-n: Excellent. I'd like to run more than one rep in dry run mode. Can you 
easily support both "-n 100" and '-n' for just one rep? We use this to do 
profiling and one rep isn't sufficient for profiling!
-i: Excellent. Should default to 'time'.
"-m: foo" Is shorthand for -Dbenchmark=foo ? I'd like to support the other form 
also, since that's how it shows up in the output.
-v: I don't think verbose is the right name for this, but I like the new option.
-s: really? I thought scores were dead.
--uploadResults: sounds good to me as a separate program. Please don't remove 
that functionality without first adding that program! I need --uploadResults 
for benchmarks run on devices.

Original comment by jessewil...@google.com on 10 Feb 2011 at 7:43

[GoogleCodeExporter]

-n: I don't think "-n 100"/"-n" is possible, no.  Is it possible that profiling 
is a fundamentally different case from a "regular" dry run and should be its 
own 'instrument' or anything?  Well, I think --dry-run-reps might be okay, 
default 1, short form -r....

-i: thanks, forgot to note a few defaults in the helptext, done now.

-m: I would really prefer not to conflate user parameters with other variables. 
 Can we lose -Dbenchmark=foo?

-v: I agree it's not a perfect fit. I need something that implies "collections 
a metric butt ton of data and includes it in the results file (that you might 
or might not be uploading)."  If you think of a good name, let me know.  The 
short form might still be -v anyway.  Or if not, then -v could perhaps be 
version.

-s: no, I wonder why it seemed dead to you... besides being just plain required 
for our internal farm infrastructure, it's a decent roll-up summary for before 
and after you make an impl change.

--uploadResults: com.google.caliper.UploadResults is in fact already there!

Original comment by kevinb@google.com on 10 Feb 2011 at 8:01

[GoogleCodeExporter]

Should "dry run" be an arg to --instrument?
  --instrument dry_run
or
  --instrument none

I like --reps 1000. Even if it's for dry_run only.

I can cope with -Dbenchmark=foo disappearing. I just thought that it made it 
easy to mechanically go from a single entry in the results to the corresponding 
parameters.
  -Dsize=100 -Dtype=ArrayList -Dbenchmark=iterate

-verbose could be "--capture" or "--full-log" ?

Original comment by jessewil...@google.com on 10 Feb 2011 at 8:07

[GoogleCodeExporter]

I'm thinking as we add more kinds of 'instruments', the idea of a dry run could 
apply to various of them.  So, we should keep it separate.

I wonder if --reps 1000 could have any use in "real life" as well, like just as 
a hint of some kind... but I'm coming up empty.

--firehose-log? :)

Original comment by kevinb@google.com on 10 Feb 2011 at 8:14

[GoogleCodeExporter]

New info on how we need to revamp JRE paths and JVM arguments.

Sometimes you want something like

-j jdk1.5,jdk1.6 -Jmemory=-Xmx32m,-Xmx128m

Because you want caliper to do its usual thing of trying all combinations.

But often there are different parameters that make sense for different JDKs. 
e.g, 

- 32-bit with -server
- 32-bit with -client
- 64-bit with -XX:UseCompressedOops
- 64-bit (regular)

What I think makes the most sense is to allow the user to define an arbitrary 
list of JVM execution strings in their caliperrc:

jvm.jdk6_32_client=/path/to/jdk1.6/bin/java -client
jvm.jdk6_32_server=/path/to/jdk1.6/bin/java -server
jvm.jdk6_64_compressed=/path/to/jdk1.6/bin/java -XX:UseCompressedOops
jvm.jdk6_64=/path/to/jdk1.6/bin/java

Then run with -j jdk6_32_client,jdk6_32_server,jdk6_64_compressed,jdk6_64

The user's assignment of a logical name to each one works out really well for 
us, because now they can both use it on their command lines and also have it 
appear under that name in their benchmark results page.  It also keeps the 
command line from being a crazy mess.  Also the fact that the space character 
has to be treated specially right now means trouble down the line when users 
want to run a benchmark on a windows machine.

These mappings belong in caliperrc because most users will want to set them 
once and rarely change them, reusing them many times.

So basically the contents of -j are a comma-separated list where each token is 
first checked to see if it has a corresponding "jvm.*" mapping in caliperrc, 
then if not, is treated as a directory name of a JRE installation. You can't 
embed args straight in it.

Original comment by kevinb@google.com on 25 Feb 2011 at 12:28

[GoogleCodeExporter]

I like the -j args in caliperrc. But please don't prefix the setting with 
'jvm.'; Caliper runs on other kinds of VM! I prefer this approach over having a 
JRE base and a relative path.

Original comment by jessewil...@google.com on 25 Feb 2011 at 6:10

[GoogleCodeExporter]

"vm" then?

To be clear, whenever a string given in the comma-separated list is not found 
in the caliperrc, then it would still be treated as a directory name relative 
to a base path -- that base path could be moved to another caliperrc setting 
instead of command-line, though. That probably makes more sense.

Original comment by kevinb@google.com on 25 Feb 2011 at 2:46

[GoogleCodeExporter]

Now I have, in .caliperrc

vm.baseDirectory=/usr/local/vms
vm.commonArguments=...
vm.detailArguments=...

vm.alias.jdk64compressed=jdk-64 -XX:+UseCompressedOops
etc.

And the command-line option is --vm, or -m for short.

Original comment by kevinb@google.com on 6 Mar 2011 at 3:58

[GoogleCodeExporter]

Original comment by kevinb@google.com on 19 Mar 2011 at 2:59

• Changed state: Started

[GoogleCodeExporter]

Original comment by kevinb@google.com on 19 Mar 2011 at 3:06

• Added labels: Type-Enhancement

[GoogleCodeExporter]

Original comment by kevinb@google.com on 8 Sep 2011 at 11:10

• Changed state: Fixed