Maven plugin for caliper

[GoogleCodeExporter]

It would be nice if caliper could be run via maven. I have something in my mind 
like:

<plugin>
  <groupId>com.google.caliper</groupId>
  <artifactId>caliper-maven-plugin</artifactId>
  <version>${caliper.version}</version>

  <!-- This should be default -->
  <configuration>
    <includes>
      <include>**/*Benchmark.java</include>
    <includes>
  </configuration>
</plugin>

Original issue reported on code.google.com by nepomuk....@gmail.com on 11 Nov 2012 at 10:42

[GoogleCodeExporter]

That _would_ be nice.  The top priority right now is getting v1.0 released, so 
it probably won't show up anytime soon, but would definitely be nice to have 
afterwards.

Original comment by gak@google.com on 12 Nov 2012 at 5:04

• Changed state: Accepted
• Added labels: Milestone-Post-1.0, Priority-Low, Type-Enhancement
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

Cool. Meanwhile I will try something on my own and may open a pull request, if 
the results are presentable.

Original comment by nepomuk....@gmail.com on 14 Nov 2012 at 1:18

[GoogleCodeExporter]

It can be run by maven with the exec plugin:

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>exec-maven-plugin</artifactId>
  <version>1.2.1</version>

  <configuration>
    <executable>java</executable>
    <arguments>
      <argument>-classpath</argument>
      <classpath />
      <argument>com.google.caliper.Runner</argument>
      <argument>${benchmark}</argument>
      <argument>${parameters}</argument>
    </arguments>
  </configuration>
</plugin>

It would be nice though if such code was put into an example module with the 
Tutorial class

Original comment by krzywick...@gmail.com on 20 Nov 2012 at 10:08

[GoogleCodeExporter]

@krzywicki-grant this POM snippet doesn't seem to work:

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2.1:java 
(default-cli) on project string-benchmarks: Unable to parse configuration of 
mojo org.codehaus.mojo:exec-maven-plugin:1.2.1:java: Cannot assign 
configuration values to array of type java.lang.String: [-classpath, Classpath 
{}, com.google.caliper.Runner, StringBenchmarks] -> [Help 1]

Original comment by dsero...@gmail.com on 8 Apr 2013 at 6:13

[GoogleCodeExporter]

FWIW, there is a patch in the works that should allow this type of invocation 
to be much easier.  I'll update this bug when it is pushed.

Original comment by gak@google.com on 8 Apr 2013 at 6:15

[GoogleCodeExporter]

I've put together a Maven plugin - feedback is more than welcome! 
https://github.com/atychyna/caliper-maven-plugin

Original comment by anton.ty...@gmail.com on 14 May 2013 at 8:49

[GoogleCodeExporter]

After a quick glance, this looks very cool.  A few notes:
  - is it possible to do something like src/benchmarks/java rather than the glob?
  - internally, we run all of our benchmarks with --dry-run as part of our testing process just to verify that we haven't caused any obvious issues with the benchmark.  could that goal be added and bound automatically?
  - i'm wary about binding actual runs to the verify phase since benchmarks can take a very long time. is that the default behavior or just an option?
  - i haven't tried it out, but this looks great.  thanks!

Original comment by gak@google.com on 14 May 2013 at 3:40

[GoogleCodeExporter]

@gak answering your question:

 > - is it possible to do something like src/benchmarks/java rather than the glob?

Plugin scans one directory looking for benchmarks, by default it's 
target/classes (can be changed with config param 'benchmarkClassesDirectory'). 
It's possible however to add another source directory with help of 
http://mojo.codehaus.org/build-helper-maven-plugin/add-source-mojo.html

 > - internally, we run all of our benchmarks with --dry-run as part of our testing process just to verify that we haven't caused any obvious issues with the benchmark.  could that goal be added and bound automatically?

Plugin has 'dryRun' config param. Basically it supports (almost) all  Caliper's 
command line params.

 > - i'm wary about binding actual runs to the verify phase since benchmarks can take a very long time. is that the default behavior or just an option?

Plugin needs to be binded to some lifecycle phase anyway in order to be 
executed as a part of a build. Verify seemed like a sensible default to me.  
It's just  the value Maven will use in case user dosen't explicitly specify 
<phase> option.

Original comment by anton.ty...@gmail.com on 14 May 2013 at 7:40

[GoogleCodeExporter]

This is great. Will you push it to maven central or wait until it's merged into 
the caliper project?

Original comment by nepomuk....@gmail.com on 14 May 2013 at 9:35

[GoogleCodeExporter]

If Caliper's team wants to adopt it I have no problems with that (BTW I'm also 
fine with changing groupId, copyright headers etc.). Otherwise I guess it can 
live on its own. As for uploading to central - plugin needs to be released 
first. Central repo won't accept snapshot artifacts. I can release it as soon 
as Caliper itself is released and we settle on proper groupId, package names 
etc.

In the meantime it'll be good to know if anyone has actually tried it out yet ;)

Original comment by anton.ty...@gmail.com on 15 May 2013 at 2:29

[GoogleCodeExporter]

Why is the default directory target/classes instead of target/test-classes
IMHO benchmarks should not be in productive code?

What's nice (and creepy somehow) the automatic upload to appspot. The plain run 
works fine for me (besides the little problem I had with main/test source 
folders).

Great job!

Original comment by nepomuk....@gmail.com on 17 May 2013 at 8:48

[GoogleCodeExporter]

@nepomuk i agree, it might be confusing - on the other hand putting benchmarks 
in src/test might confuse people as well (they are not tests after all). I 
think what gak has proposed is a nice compromise: put them into src/benchmarks 
and add it as a source folder to the build (btw you can use 
'benchmarkClassesDirectory' option but i think you've figured that out yourself 
already).

Original comment by anton.ty...@gmail.com on 17 May 2013 at 9:49

[GoogleCodeExporter]

I've been using the maven plugin to get some of our benchmarks running under 
linux. 

My biggest grumble is with the fact that the plugin doesn't support having 
benchmarks as "tests". I don't like adding caliper it self as a compile time 
dependency. (Though thinking about it a bit more it could probably be dealt 
with by using profiles. I need to go experiment)

The the other grumbles are that there is no reporting if the Benchmark doesn't 
complete (I've seen a few problems with them bombing out and the plugin 
continues on happily and mark the build as green). 

The reason I noticed the failed runs was having to dig through the console 
output to find the run URL's. It would be much more helpful if the plugin could 
collect and output the run urls to a text file that could be picked up by 
jenkins et al.

Original comment by sam.bar...@mahifx.com on 14 Jun 2013 at 3:52