How can I run benchmarks on cold start?

[dmitry-stakhov]

In the next snippet the code I want to benchmark is called only on the first benchmark iteration. How can I make it to be called on each iteration on JVM and native targets?

object Init {
    val value = doSomeInit().apply {
        println("INIT")
    }

    private fun doSomeInit() = 1 + 1
}

@State(Scope.Benchmark)
class NetworkingBenchmark {
    @Benchmark
    fun runBenchmark() {
        Init.value
    }
}

[fzhinkin]

@dmitry-stakhov, on JVM, you can use JMH's SingleShotTime mode (either by manually supplying -bm ss option to the generated jar, or by using corresponding mode value in BenchmarkMode annotation). For Native, however, there's no such a benchmarking mode.

[akiya-nagatsuka]

@fzhinkin I tried this, but still init is called only once.

// in test gradle module
object Test {
    val test = 1.apply {
        println("INIT")
    }
}

// in benchmark Gradle module
import kotlinx.benchmark.Benchmark
import kotlinx.benchmark.BenchmarkMode
import kotlinx.benchmark.Scope
import kotlinx.benchmark.State
import org.openjdk.jmh.annotations.Mode.SingleShotTime

@State(Scope.Benchmark)
@BenchmarkMode(SingleShotTime)
class ObjectBenchmark {
    @Benchmark
    fun objectBenchmark() {
        val a = Test.test
    }
}

// Output
INIT
Warm-up 1: ≈ 10⁻⁶ ms/op
Warm-up 2: ≈ 10⁻⁶ ms/op
Iteration 1: ≈ 10⁻⁷ ms/op
Iteration 2: ≈ 10⁻⁷ ms/op
Iteration 3: ≈ 10⁻⁷ ms/op

[fzhinkin]

@akiya-nagatsuka, given multiple warmup and measurement iterations, it seems like SingleShotTime didn't applied, for some reason.

I took the benchmark as it is and ss-mode worked as expected:

$ ./gradlew mainBenchmark
....
> Task :mainBenchmark
Running 'main' benchmarks for 'main'

… org.example.ObjectBenchmark.objectBenchmark

INIT
Iteration 1: 0.001 s/op

  Success: 

main summary:
Benchmark                        Mode  Cnt  Score   Error  Units
ObjectBenchmark.objectBenchmark    ss       0.001           s/op

Maybe you can share a project so we could figure out why the benchmark does not behave as it should for you?

[dmitry-stakhov]

@fzhinkin Please check the reproducer BenchmarkReproducer.zip

[fzhinkin]

@dmitry-stakhov

Following config overrides benchmark's settings:

        named("main") {
            mode = "AverageTime"
            warmups = 2
            iterations = 30
            iterationTime = 50
            iterationTimeUnit = "ms"
            outputTimeUnit = "ms"
        }

If you need to run benchmarks via Gradle tasks, (as a workaround) you can add a configuration like that:

        create("coldStart") {
            advanced("jvmForks", 10) // or any other value
            include("benchmarks\\.Benchmarks")
            outputTimeUnit = "ms"
        }

and it'll do the trick:

$ ./gradlew coldStartBenchmark
...
… benchmarks.Benchmarks.fetchGlobalConfigBenchmark

Init
Iteration 1: 1.041 ms/op

Init
Iteration 1: 0.699 ms/op

Init
Iteration 1: 0.785 ms/op

Init
Iteration 1: 0.776 ms/op

Init
Iteration 1: 0.677 ms/op

Init
Iteration 1: 1.101 ms/op

Init
Iteration 1: 0.759 ms/op

Init
Iteration 1: 0.649 ms/op

Init
Iteration 1: 0.779 ms/op

Init
Iteration 1: 0.803 ms/op

  Success: N = 10
  mean =      0.807 ±(99.9%) 0.225 ms/op

  Histogram, ms/op:
    [0.600, 0.650) = 1 
    [0.650, 0.700) = 2 
    [0.700, 0.750) = 0 
    [0.750, 0.800) = 4 
    [0.800, 0.850) = 1 
    [0.850, 0.900) = 0 
    [0.900, 0.950) = 0 
    [0.950, 1.000) = 0 
    [1.000, 1.050) = 1 
    [1.050, 1.100) = 0 
    [1.100, 1.150) = 1 
    [1.150, 1.200) = 0 

  Percentiles, ms/op:
      p(0.0000) =      0.649 ms/op
     p(50.0000) =      0.777 ms/op
     p(90.0000) =      1.095 ms/op
     p(95.0000) =      1.101 ms/op
     p(99.0000) =      1.101 ms/op
     p(99.9000) =      1.101 ms/op
     p(99.9900) =      1.101 ms/op
     p(99.9990) =      1.101 ms/op
     p(99.9999) =      1.101 ms/op
    p(100.0000) =      1.101 ms/op

jvm summary:
Benchmark                              Mode  Cnt  Score   Error  Units
Benchmarks.fetchGlobalConfigBenchmark    ss   10  0.807 ± 0.225  ms/op

[fzhinkin]

But in general, it might be worth providing a single-shot / cold-start benchmarking mode for all supported targets in the future. (CC @qurbonzoda)