search

toshsan / caliper

Automatically exported from code.google.com/p/caliper
Apache License 2.0
0 stars 0 forks source link

instrument should be reported under consistent name #202

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
instrument has both "config name" and "class name"

only config name seems to be unique. I suggest use unique name throughout

##################################

# Instrument "micro"

# You probably don't want to change this
instrument.micro.class=com.google.caliper.runner.MicrobenchmarkInstrument

##################################

Experiment selection: 
  Instruments:   [AllocationInstrument, MicrobenchmarkInstrument]
  User parameters:   {arguments=[1, 2, 3, 4, 5, 6], strategy=[VARARGS, EXPANSION]}
  Virtual machines:  [default]
  Selection type:    Full cartesian product

##################################

Original issue reported on code.google.com by Andrei.Pozolotin on 17 Jan 2013 at 6:03

GoogleCodeExporter commented 9 years ago 
I think you may be misinterpreting how these identifiers are used.  In the 
config file, "micro" is really only used to group properties.  The class name 
points to the implementation.  You theoretically could change out the 
implementation of the instrument, but that's really not something that should 
ever be done except for maybe development.

The reason "micro" exists is to be able to specify configuration using 
something more sensible than 
instrument.com.google.caliper.runner.MicrobenchmarkInstrument.options.whatever. 

Now, in the output, we probably could use "micro", et. al.  We do something 
similar for VMs, but that was really only because there's not a good way to 
print something succinct for "64 bit Java 7 with a 5GB heap".  But, the choice 
to stick with the class name was because that's what gets reported in the 
output, so it seemed like it was better to be consistent between the output and 
the results than between the output and the config file since the config file 
is somewhat rarely edited. 

For the time being, I'm going to mark this WontFix, but feel free to follow up 
if there's something here that I'm missing.

Original comment by gak@google.com on 17 Jan 2013 at 8:21

GoogleCodeExporter commented 9 years ago

Original comment by gak@google.com on 17 Jan 2013 at 8:22

GoogleCodeExporter commented 9 years ago 
in my case I have multiple "micro" : "micro-this" and "micro-that"
they all use the same class, but differ in options

Original comment by Andrei.Pozolotin on 17 Jan 2013 at 11:32

GoogleCodeExporter commented 9 years ago 
I do think I mistakenly output the wrong name there.  I think I'd rather see 
"Instruments:  [allocation, micro]".

Original comment by kevinb@google.com on 18 Jan 2013 at 3:03

GoogleCodeExporter commented 9 years ago 
I could see the argument, but it's a little weird for me to output in terms of 
the value that most users will never see instead of the value that will show up 
in every dataset.  Obviously they're quite similar, so it's not like it will 
really matter one way or the other, but I'm still feeling a little underwhelmed 
about wanting to change behavior to address what seems like a bit of a 
non-issue.

Since there seems to be at least _some_ debate about it, I'll move it back to 
New.

Original comment by gak@google.com on 18 Jan 2013 at 5:57

GoogleCodeExporter commented 9 years ago 
To your first sentence: *exactly*!  So let's start using 'micro' everywhere 
instead of saddling users with the implementation detail of 
MicrobenchmarkInstrument.  'micro' is already used by the --instrument 
command-line flag.  Its whole *intent* was to be the user-visible name.

Original comment by kevinb@google.com on 18 Jan 2013 at 2:57

GoogleCodeExporter commented 9 years ago 
Except, just like VM names, they're arbitrary and only have meaning in the 
context of a given user's config file.  As soon as any of the options change 
(worst of all the class), the identifier is, at best, misleading.

Original comment by gak@google.com on 18 Jan 2013 at 5:52

GoogleCodeExporter commented 9 years ago 
since people want to share benchmarks, there has to be a way to ensure they are 
running with same settings/options. how will you do that?

Original comment by Andrei.Pozolotin on 18 Jan 2013 at 11:19

GoogleCodeExporter commented 9 years ago 
OK, this is starting to feel like a 20-minute conversation we need to have.

Original comment by kevinb@google.com on 19 Jan 2013 at 5:45

GoogleCodeExporter commented 9 years ago 
@Andrei, that's exactly what the UI _does_ do.  It reports *all* instrument 
options, JVM options, etc.  What it doesn't do is report that User A ran with a 
configuration called "micro" that had a timing interval of 500ms and User B ran 
with a configuration called "micro" that had a timing interval of 1s.  That 
seems misleading at best.

Here's an example of the relevant portion of the JSON output:
"instrumentSpec": {
  "className": "com.google.caliper.runner.MicrobenchmarkInstrument",
  "options": {
    "gcBeforeEach": "true",
    "measurements": "9",
    "timingInterval": "500ms",
    "warmup": "10s"
  }
}

Original comment by gak@google.com on 21 Jan 2013 at 8:53

GoogleCodeExporter commented 9 years ago 
@Gregory great, got it. 
BTW, I am not able to publish anything with code from master.
can you share new uri or code / config example please?

Original comment by Andrei.Pozolotin on 21 Jan 2013 at 9:12

GoogleCodeExporter commented 9 years ago 
The new webapp that understands the JSON has not been deployed.  There will be 
an announcement when it's ready.

Original comment by gak@google.com on 21 Jan 2013 at 9:16

GoogleCodeExporter commented 9 years ago 
The instrument is now reported using it's "config name" on the console.

Original comment by gak@google.com on 5 Feb 2013 at 7:59

GoogleCodeExporter commented 9 years ago 
thank you.

Original comment by Andrei.Pozolotin on 10 Feb 2013 at 2:20