Multiple transformation issue with JMH benchmarks

[kavehshahedi]

Hello,

I have built a Java agent that tries to instrument some methods in the main program by running the JMH benchmarks. However, the nature of the JMH structure has generated two issues for having a precise instrumentation information;

1. Warmup Ignorance: Since JMH benchmarks need to have prior warmups, the agent needs to skip the code transformations since no instrumentation should be done. As we don't have access to the benchmark classes themselves, how can we skip the transformation during the warmup step?
2. Multiple Transformations Due to Forking: JMH benchmarking provides an option to have multiple forks of a benchmark execution along with multiple iterations in each fork. However, the issue is that for each new fork, the agent re-transforms the benchmark jar. Hence, the re-transformation and re-compilation have a significant overhead for cases with a high number of forks. Is there a way to fix this issue?

This is the command I use for running the benchmarks:

java -javaagent:instrumentor.jar -jar target/benchmarks.jar -f 5 -i 5 -wi 3 BENCHMARK_NAME

P.S.: The instrumentation agent is just a simple byte-buddy transformation tool. Nothing complex or sophisticated for transforming.

Thanks

[raphw]

Unfortunately, I would not know. Byte Buddy creates agents without regards to tools. Maybe you could hear on the JMH mailing list how this can be solved for Java agents in general?

As for the warmup: maybe you can skip instrumentation using a flag? But this would defeat the purpose of the warmup. As with forking: Maybe it would be better to transform the jar statically using a build tool? Byte Buddy offers support for this, too.