How to benchmark jdk.incubator (jvm)

[quickstep24]

I am trying to benchmark code that uses jdk.incubator.vector methods, but this raises an error in :jvmBenchmarkGenerate:

Generation of JMH bytecode failed with 1errors:
  - Annotation generator had thrown the exception.
java.lang.NoClassDefFoundError: jdk/incubator/vector/VectorSpecies

A minimal benchmark code is

@State(Scope.Benchmark)
open class MyBenchmark {
    val species: VectorSpecies<Long> = LongVector.SPECIES_PREFERRED
    @Benchmark fun test() = species.length()
}

I have tried adding @Fork(jvmArgs = ["--add-modules", "jdk.incubator.vector"]) and also added to build.gradle.kts:

tasks.withType<JavaExec> {
    if (name == "jvmBenchmark") {
        jvmArgs(listOf("--add-modules", "jdk.incubator.vector"))
    }
}

but without success.

[sureshg]

Hi, could you able to solve this issue (for multiplatform setup)?.. i am slo facing the same issue when using vector APIs with jdk 21.

[quickstep24]

No, in the end I used conventional time measurement (after some warming-up loops). My test cases were simple Boolean operations on large bitsets. I could not identify a performance gain, so I went back to conventional code. However, results may depend on your CPU architecture and I used an older jdk version.

[sureshg]

Thanks...eventually i could able to run with @Fork(value = 1, jvmArgsAppend = ["--add-modules=jdk.incubator.vector"]) and some kotlin mpp compileJava and test configurations . Wish the framework had builtin support to configure more JMH parameters instead of hardcoding it in the Benchmark tests.

Edit: sorry, no this is not working as expected :/

[fxshlein]

I'm not running a multiplatform setup, but I think the underlying problem and task are the same. The second JVM that does not have jdk.incubator.vector loaded is launched here. Since no jvm arguments are specified, the fork that runs the annotation processing is executed without the extra module.

I was able to temporarily circumvent this by removing the existing action from the task, and adding my own copy, which includes the additional JVM argument. In my build.gradle.kts:

afterEvaluate {
    val workerExecutor = serviceOf<WorkerExecutor>()

    @OptIn(KotlinxBenchmarkPluginInternalApi::class)
    tasks.withType<JmhBytecodeGeneratorTask> {
        // Remove the existing @TaskAction
        actions.clear()

        // Custom task action, with vector module
        doLast {
            val workQueue = workerExecutor.processIsolation {
                classpath.setFrom(runtimeClasspath.files)
                if (executableProvider.isPresent) {
                    forkOptions.executable = executableProvider.get()
                }
                // Add required argument:
                forkOptions.jvmArgs = listOf("--add-modules", "jdk.incubator.vector")
            }

            workQueue.submit(JmhBytecodeGeneratorWorker::class.java) {
                inputClasses.setFrom(inputClassesDirs.files)
                inputClasspath.setFrom(inputCompileClasspath.files)
                outputSourceDirectory.set(outputSourcesDir)
                outputResourceDirectory.set(outputResourcesDir)
            }

            workQueue.await()
        }
    }
}

The proper solution to this would probably be to either allow specifying JVM arguments as task inputs, or even better, to make the task extend JavaForkOptions. AFAIK, that's how gradle tasks that spawn a java fork are marked conventionally. This then also supports things like debugging in the JMH generator from IntelliJ. JavaForkOptions::copyTo can then be used to copy the settings of the task to workerSpec.forkOptions.