search

toshsan / caliper

Automatically exported from code.google.com/p/caliper
Apache License 2.0
0 stars 0 forks source link

caliper measure pattern should support network bandwidth benchmarks #186

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago 
thank your for the caliper - great idea.

I have tried to use caliper for network bandwidth benchmarks with netty
https://github.com/netty/netty

in my experience current design pattern of caliper is unusable for this due to:

1) fixed reps invocation pattern: 
does not suit well for non-blocking NIO paradigm;

2) only 3 dimensions can be reported/analysed: 
run time, instance count, memory heap size;

re: 1) I suggest: instead of timing each run with fixed reps invocations, 
have a fixed run time, and do run performance parameter sampling during the run,
like metrics does http://metrics.codahale.com/

re: 2) I suggest: allow for more measures and make those measures more 
generic/configurable.

here is example how I hacked caliper v 0.5 to resolve both 1) and 2):

custom caliper runner
https://github.com/barchart/netty-udt/blob/master/transport-udt/src/test/java/io
/netty/transport/udt/util/CaliperRunner.java

custom caliper benchmark
https://github.com/barchart/netty-udt/blob/master/transport-udt/src/test/java/io
/netty/transport/udt/util/CaliperBench.java

custom caliper + metrics measure
https://github.com/barchart/netty-udt/blob/master/transport-udt/src/test/java/io
/netty/transport/udt/util/CaliperMeasure.java

cheers.

Original issue reported on code.google.com by Andrei.Pozolotin on 7 Jan 2013 at 5:17

GoogleCodeExporter commented 9 years ago 
I think that this is an accurate bug report in that the microbenchmark 
instrument in either Caliper 0.5 or 1.0 are just not set up to do the type of 
benchmark that you're trying to do.  Your suggestion seems like a reasonable 
alternate instrument, but couldn't be a replacement for a variety of reasons.

#2 is already solved in Caliper 1.0.  Coming very, very soon. :-)

Original comment by gak@google.com on 7 Jan 2013 at 6:06

GoogleCodeExporter commented 9 years ago 
Gregory:

thank you for considering this.

I cloned 1.0-SNAPSHOT - looks great already! :-)

I am not asking for a replacement but for addition.

Or, may be I am asking for making caliper into more lego-like open tool set
so instruments that I am looking for would be easier for me to build on top of 
it.

Cheers.

Andrei.

Original comment by Andrei.Pozolotin on 8 Jan 2013 at 4:24

GoogleCodeExporter commented 9 years ago

Original comment by gak@google.com on 10 Jan 2013 at 6:26