slachiewicz / caliper

Automatically exported from code.google.com/p/caliper
Apache License 2.0
0 stars 0 forks source link

Memory measurement just plain fails #150

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago
I ran this in Eclipse with --measureMemory, and got the following:

 0% Scenario{vm=java, trial=0, benchmark=AllocateMultiset}  Failed to execute java -javaagent:null -cp /home/lowasser/workspace/Benchmarks/bin:/home/lowasser/git/caliper/caliper/target/classes:/home/lowasser/git/caliper/caliper/target/test-classes:/home/lowasser/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/lowasser/.m2/repository/com/google/code/gson/gson/1.7.1/gson-1.7.1.jar:/home/lowasser/.m2/repository/com/google/guava/guava/11.0.1/guava-11.0.1.jar:/home/lowasser/.m2/repository/com/google/code/java-allocation-instrumenter/java-allocation-instrumenter/2.0/java-allocation-instrumenter-2.0.jar:/home/lowasser/.m2/repository/asm/asm/3.3.1/asm-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-analysis/3.3.1/asm-analysis-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-commons/3.3.1/asm-commons-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-tree/3.3.1/asm-tree-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-util/3.3.1/asm-util-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-xml/3.3.1/asm-xml-3.3.1.jar:/home/lowasser/.m2/repository/junit/junit/3.8.2/junit-3.8.2.jar:/home/lowasser/git/wassermanlouis-guava/guava/target/classes:/home/lowasser/Downloads/jsr166e.jar:/home/lowasser/workspace/Experimental/target/classes:/home/lowasser/Downloads/object-explorer.jar:/home/lowasser/Downloads/MemoryMeasurer-0.1.zip:/home/lowasser/git/wassermanlouis-guava/guava-testlib/target/classes com.google.caliper.InProcessRunner --warmupMillis 3000 --runMillis 1000 --measurementType INSTANCE --marker //ZxJ/ -Dbenchmark=AllocateMultiset MultisetAllocationBenchmark
Error opening zip file or JAR manifest missing : null
Error occurred during initialization of VM
agent library failed to init: instrument

An exception was thrown from the benchmark code.
com.google.caliper.ConfigurationException: Failed to execute java 
-javaagent:null -cp 
/home/lowasser/workspace/Benchmarks/bin:/home/lowasser/git/caliper/caliper/targe
t/classes:/home/lowasser/git/caliper/caliper/target/test-classes:/home/lowasser/
.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/lowa
sser/.m2/repository/com/google/code/gson/gson/1.7.1/gson-1.7.1.jar:/home/lowasse
r/.m2/repository/com/google/guava/guava/11.0.1/guava-11.0.1.jar:/home/lowasser/.
m2/repository/com/google/code/java-allocation-instrumenter/java-allocation-instr
umenter/2.0/java-allocation-instrumenter-2.0.jar:/home/lowasser/.m2/repository/a
sm/asm/3.3.1/asm-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-analysis/3.3.1/
asm-analysis-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-commons/3.3.1/asm-c
ommons-3.3.1.jar:/home/lowasser/.m2/repository/asm/asm-tree/3.3.1/asm-tree-3.3.1
.jar:/home/lowasser/.m2/repository/asm/asm-util/3.3.1/asm-util-3.3.1.jar:/home/l
owasser/.m2/repository/asm/asm-xml/3.3.1/asm-xml-3.3.1.jar:/home/lowasser/.m2/re
pository/junit/junit/3.8.2/junit-3.8.2.jar:/home/lowasser/git/wassermanlouis-gua
va/guava/target/classes:/home/lowasser/Downloads/jsr166e.jar:/home/lowasser/work
space/Experimental/target/classes:/home/lowasser/Downloads/object-explorer.jar:/
home/lowasser/Downloads/MemoryMeasurer-0.1.zip:/home/lowasser/git/wassermanlouis
-guava/guava-testlib/target/classes com.google.caliper.InProcessRunner 
--warmupMillis 3000 --runMillis 1000 --measurementType INSTANCE --marker //ZxJ/ 
-Dbenchmark=AllocateMultiset MultisetAllocationBenchmark
    at com.google.caliper.Runner.measure(Runner.java:310)
    at com.google.caliper.Runner.runScenario(Runner.java:237)
    at com.google.caliper.Runner.runOutOfProcess(Runner.java:379)
    at com.google.caliper.Runner.run(Runner.java:98)
    at com.google.caliper.Runner.main(Runner.java:424)
    at com.google.caliper.Runner.main(Runner.java:441)
    at MultisetAllocationBenchmark.main(MultisetAllocationBenchmark.java:29)

...I guess there's something with the javaagent?  I've had to do explicit 
command line things when using other memory measurement tools in Java.

Original issue reported on code.google.com by wasserman.louis on 16 Feb 2012 at 10:03

Attachments:

GoogleCodeExporter commented 9 years ago
We need to either fix or remove this functionality for the 0.5 release.

Original comment by fry@google.com on 16 Feb 2012 at 10:39

GoogleCodeExporter commented 9 years ago
I'd guess you don't have ALLOCATION_JAR set. And this isn't yet resolved:  
http://code.google.com/p/caliper/issues/detail?id=114/

Original comment by brianfromoregon on 17 Feb 2012 at 1:11

GoogleCodeExporter commented 9 years ago
In particular, there's no indication anywhere what ALLOCATION_JAR is supposed 
to be set to.

Original comment by wasserman.louis on 17 Feb 2012 at 1:57

GoogleCodeExporter commented 9 years ago
Yeah, the goal here should be for users to not have to set anything special.

Original comment by kevinb@google.com on 17 Feb 2012 at 5:02

GoogleCodeExporter commented 9 years ago
Kevin, what do you recommend for the 0.5 release? Should this functionality 
simply be removed?

Original comment by fry@google.com on 21 Feb 2012 at 7:53

GoogleCodeExporter commented 9 years ago
Has this been resolved. 
Using 0.5 the allocation.jar found here : 
http://code.google.com/p/java-allocation-instrumenter/ and asm-3.0.jar (if its 
needed), but still get the same error

Original comment by BenAbout...@gmail.com on 21 Mar 2012 at 1:42

GoogleCodeExporter commented 9 years ago
Actually when passing defining the ALLOCATION_JAR env I have another error:

Failed to find Premain-Class manifest attribute in 
/Users/brice/.m2/repository/com/google/code/java-allocation-instrumenter/java-al
location-instrumenter/2.0/java-allocation-instrumenter-2.0.jar
Error occurred during initialization of VM
agent library failed to init: instrument

The allocation instrumenter project should update it's maven build to include 
this manifest entry.

Modifying the MANIFEST.MF of the allocation jar, got me further, still there is 
an instrumentation error : 

Class loading breakage: Will not be able to instrument JDK classes
[Ljava/lang/Object;
[Lcom/google/caliper/MeasurementType;
[Lcom/google/common/collect/AbstractIterator$State;
[Lcom/google/common/base/AbstractIterator$State;
[Lexo2/Solution2Benchmark$Distribution;
[Ljava/lang/Object;
starting Scenario{vm=java, trial=0, benchmark=Solution2a, K=1, 
distribution=SAWTOOTH, length=10}
[caliper] [starting measured section]
[caliper] see first run for list of allocations
[caliper] [done measured section]
[caliper] 2 instance(s) allocated per rep
[caliper] 0 out of thread instance(s) allocated in 1 reps
[caliper] [starting measured section]
[caliper] see first run for list of allocations
[caliper] [done measured section]
[caliper] 2 instance(s) allocated in 1 reps
[caliper] [starting measured section]
[caliper] see first run for list of allocations
[caliper] [done measured section]
[caliper] 3 instance(s) allocated in 2 reps
[caliper] ignoring 1 allocation(s) per measurement as overhead
[caliper] [starting measured section]
[caliper] allocating exo2/Solution2a with size 16 bytes
[caliper] [done measured section]
[caliper] 1 instance(s) allocated per rep
[caliper] 0 out of thread instance(s) allocated in 1 reps
[caliper] [starting measured section]
[caliper] see first run for list of allocations
[caliper] [done measured section]
[caliper] 1 instance(s) allocated per rep
[caliper] 0 out of thread instance(s) allocated in 2 reps
[caliper] [starting measured section]
[caliper] see first run for list of allocations
[caliper] [done measured section]
[caliper] 1 instance(s) allocated per rep
[caliper] 0 out of thread instance(s) allocated in 3 reps
[caliper] [starting measured section]
[caliper] see first run for list of allocations
[caliper] [done measured section]
[caliper] 1 instance(s) allocated per rep
[caliper] 0 out of thread instance(s) allocated in 4 reps
[Lcom/google/gson/LongSerializationPolicy;
[Ljava/lang/reflect/Type;
Exception in thread "main" java.lang.VerifyError: (class: 
com/google/gson/internal/$Gson$Types, method: resolve signature: 
(Ljava/lang/reflect/Type;Ljava/lang/Class;Ljava/lang/reflect/Type;)Ljava/lang/re
flect/Type;) Inconsistent stack height 1 != 0
    at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:98)
    at com.google.gson.JsonSerializationContextDefault.serialize(JsonSerializationContextDefault.java:62)
    at com.google.gson.JsonSerializationContextDefault.serialize(JsonSerializationContextDefault.java:53)
    at com.google.gson.Gson.toJsonTree(Gson.java:220)
    at com.google.gson.Gson.toJson(Gson.java:260)
    at com.google.gson.Gson.toJson(Gson.java:240)
    at com.google.caliper.Json.measurementSetToJson(Json.java:66)
    at com.google.caliper.InProcessRunner.run(InProcessRunner.java:50)
    at com.google.caliper.InProcessRunner.main(InProcessRunner.java:103)

Note however that I'm using version 2.0 that is downloaded as a dependency of 
caliper 0.5-rc1. A quick look at the artifact status, showed me that this 
allocation jar was published on april 2011, however the project site shows 
there have been development earlier in 2012. I'm feeling almost lucky, I 
installed the 2.1-SNAPSHOT artefact (with MANIFEST patching), and voila no more 
error !

I guess I'm not so lucky right now, I got 0s everywhere !

 0% Scenario{vm=java, trial=0, benchmark=Solution2a, K=1, distribution=SAWTOOTH, length=10} 116.46 ns; σ=0.41 ns @ 3 trials, allocated 0 instances for a total of 0B
 1% Scenario{vm=java, trial=0, benchmark=Solution2b, K=1, distribution=SAWTOOTH, length=10} 419.19 ns; σ=1.20 ns @ 3 trials, allocated 0 instances for a total of 0B
 3% Scenario{vm=java, trial=0, benchmark=Solution2a, K=6, distribution=SAWTOOTH, length=10} 115.57 ns; σ=0.67 ns @ 3 trials, allocated 0 instances for a total of 0B
 4% Scenario{vm=java, trial=0, benchmark=Solution2b, K=6, distribution=SAWTOOTH, length=10} 350.62 ns; σ=0.85 ns @ 3 trials, allocated 0 instances for a total of 0B
 5% Scenario{vm=java, trial=0, benchmark=Solution2a, K=25, distribution=SAWTOOTH, length=10} 125.71 ns; σ=0.72 ns @ 3 trials, allocated 0 instances for a total of 0B
 6% Scenario{vm=java, trial=0, benchmark=Solution2b, K=25, distribution=SAWTOOTH, length=10} 343.52 ns; σ=0.89 ns @ 3 trials, allocated 0 instances for a total of 0B
 8% Scenario{vm=java, trial=0, benchmark=Solution2a, K=7, distribution=SAWTOOTH, length=10} 116.45 ns; σ=0.35 ns @ 3 trials, allocated 0 instances for a total of 0B
 9% Scenario{vm=java, trial=0, benchmark=Solution2b, K=7, distribution=SAWTOOTH, length=10} 342.22 ns; σ=1.21 ns @ 3 trials, allocated 0 instances for a total of 0B

Original comment by brice.du...@gmail.com on 29 Aug 2012 at 5:53

GoogleCodeExporter commented 9 years ago
By the way it seems the Premain-Class entry issue had already been reported : 
http://code.google.com/p/java-allocation-instrumenter/issues/detail?id=6

Original comment by brice.du...@gmail.com on 29 Aug 2012 at 5:54

GoogleCodeExporter commented 9 years ago
[deleted comment]
GoogleCodeExporter commented 9 years ago
OK, I was misguided by the previous comment so I didn't even bother to try the 
downloadable allocation.jar.
It was a mistake as I had to debug a lot, also when I tried to build my own 
allocation jar, I used maven, but the generated jar is incorrect as well as an 
agent. so it wasn't able to instrument classes.
I finally noticed that I should instead use the ant build which is doing some 
other stuff like jarjaring, adding the correct manifest entries to the 
allocation jar.

So, now it is working as expected : 

...
45% Scenario{vm=java, trial=0, benchmark=Solution2a, K=7, distribution=RANDOM, 
length=100} 3701.99 ns; σ=22.06 ns @ 3 trials, allocated 1 instances for a 
total of 16B
46% Scenario{vm=java, trial=0, benchmark=Solution2b, K=7, distribution=RANDOM, 
length=100} 4498.57 ns; σ=45.15 ns @ 3 trials, allocated 308 instances for a 
total of 10144B
48% Scenario{vm=java, trial=0, benchmark=Solution2a, K=111111111, 
distribution=RANDOM, length=100} 3729.27 ns; σ=16.84 ns @ 3 trials, allocated 
1 instances for a total of 16B
49% Scenario{vm=java, trial=0, benchmark=Solution2b, K=111111111, 
distribution=RANDOM, length=100} 4579.87 ns; σ=8.15 ns @ 3 trials, allocated 
308 instances for a total of 10144B
...
 benchmark length distribution         K instances        B       ns linear runtime
...
olution2a  10000       RANDOM        25     1.000     16.0 24666650 
=================
Solution2a  10000       RANDOM         7     1.000     16.0 24432650 
=================
Solution2a  10000       RANDOM 111111111     1.000     16.0 24464100 
=================
Solution2b     10     SAWTOOTH         1     9.000    320.0      469 =
Solution2b     10     SAWTOOTH         6     9.000    320.0      426 =
...
Solution2b  10000       RANDOM         7 30014.000 931264.0   835850 =
Solution2b  10000       RANDOM 111111111 30013.000 931264.0   835717 =

So basically what should be done is
1. to generate a correct allocation jar, also the build systeme should only be 
based on maven or ant but not both

2. that could eventually be uploaded on maven, ensuring the name stays the 
same, it's important otherwise you'll get some signers issues

3. detect automatically the location of this jar : if this allocation jar is a 
dependency, it should be visible in the classpath, then we can use the 
getProtectionDomain().getCodeSource().getLocation() on some class we know is 
present in this allocation jar. And as a fallback report an error asking the 
user to set either a property or the environment variable ALLOCATION_JAR

Anyway thanks for the project ! It's awesome :)

Original comment by brice.du...@gmail.com on 29 Aug 2012 at 9:40

GoogleCodeExporter commented 9 years ago

Original comment by kevinb@google.com on 1 Nov 2012 at 8:32

GoogleCodeExporter commented 9 years ago

Original comment by gak@google.com on 11 Apr 2013 at 10:27