Separating source sets for benchmarks in multiplatform projects

[goncalossilva]

I've tried adding:

val commonBenchmark by creating {
    dependsOn(commonMain) // or commonTest

    dependencies {
        implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.3.1")
        implementation("com.goncalossilva:resources:0.2.1")
    }
}

And then configuring targets as documented:

benchmark {
    targets {
        register("js")
        // register("jvm")
        // ...
    }
}

But this is clearly not the right way, since nothing happens when running (“Test events were not received”) and there is this warning:

The Kotlin source set commonBenchmark was configured but not added to any Kotlin compilation. You can add a source set to a target's compilation by connecting it with the compilation's default source set using 'dependsOn'.
BUILD SUCCESSFUL in 5s
1 actionable task: 1 executed
> Task :assembleBenchmarks UP-TO-DATE
> Task :kotlinNpmCachesSetup
> Task :jsPackageJson
> Task :kotlinNodeJsSetup UP-TO-DATE
> Task :kotlinYarnSetup UP-TO-DATE
> Task :jsBenchmarkPackageJson UP-TO-DATE
> Task :rootPackageJson
> Task :kotlinNpmInstall
> Task :jsGenerateExternalsIntegrated SKIPPED
> Task :compileKotlinJs UP-TO-DATE
> Task :jsProcessResources NO-SOURCE
> Task :jsMainClasses UP-TO-DATE
> Task :jsBenchmarkGenerate UP-TO-DATE
> Task :compileBenchmarkKotlinJs NO-SOURCE
> Task :jsBenchmark SKIPPED
> Task :benchmark UP-TO-DATE
BUILD SUCCESSFUL in 723ms
9 actionable tasks: 4 executed, 5 up-to-date
23:44:03: Execution finished 'benchmark'

Could this be documented?

[ephemient]

Yeah the docs aren't explicit about how to do it, but you need to create a compilation for that sourceset.

kotlin {
    targets {
        js {
            compilations.create("bench")
        }
    }
    sourceSets {
        val commonBench by creating {
            dependsOn(commonMain)
            dependencies {
                implementation("org.jetbrains.kotlinx:kotlinx-benchmark-runtime:0.4.0")
            }
        }
        val jsBench by existing {
            dependsOn(commonBench)
            dependsOn(jsMain)
        }
    }
}
benchmark {
    targets {
        register("jsBench")
    }
}

[goncalossilva]

Thanks! I'm unable to make this work:

> Cannot add a KotlinJsIrCompilation with name 'benchmark' as a KotlinJsIrCompilation with that name already exists.

---

If I remove js { compilations.create("benchmark") }, then I get:

KotlinSourceSet with name 'jsBenchmark' not found.

---

If instead I remove register("jsBenchmark"), I get:

Caused by: java.lang.IllegalStateException: Projects must be configuring

[ephemient]

Copied more directly from my actual project which uses the name "bench", not "benchmark"; perhaps the latter conflicts with some things kotlinx-benchmark sets up itself. (You did remove register("js") though, right?)

[elect86]

A little hint (I still cant get this working though), targets in kotlin { } has to go after sourceSets { }

[ephemient]

That shouldn't be necessary. My actual project: https://github.com/ephemient/aoc2021/blob/main/kt/build.gradle.kts (contains a lot of extra stuff you don't care about, but it runs benchmarks on a single target's "bench" compilation/sourceset).

[elect86]

If I dont, I do get

KotlinSourceSet with name 'jvmBench' not found. // jsBenchmark in your (second) case

[BartArys]

For those who followed @ephemient's instructions and are getting the following gradle error:

Project#afterEvaluate(Action) on project ':rest-ktor' cannot be executed in the current context.

You'll want to change the existing to getting from:

        val jsBench by existing {
            dependsOn(commonBench)
            dependsOn(jsMain)
        }

to:

        val jsBench by getting {
            dependsOn(commonBench)
            dependsOn(jsMain)
        }

[deepak-nulu]

The following commit shows how I got it working in my multi-platform project based on the responses from @ephemient.

It is a minimal commit with just the changes. Make sure you read the README.md file in the commit, especially if you only have common benchmarks and no JVM-specific benchmarks.

https://github.com/tree-ware/tree-ware-kotlin-core/commit/25d284167ae9b0c4d55ea546d45b7e4ec395c2f1

The benchmark tasks then show up in the Gradle panel on the right (in IntelliJ IDEA) in their own category of tasks: