search

bazelbuild / rules_kotlin

Bazel rules for Kotlin
Apache License 2.0
331 stars 206 forks source link

kt_jvm_library increased memory usage with rules_kotlin v1.9.2 #1171

Closed qtica closed 3 months ago

qtica commented 4 months ago

Starting with rules_kotlin v1.9.2, I am unable to compile kotlin targets with 1.5GB of memory on my remote executors.

Prior to v1.9.2, I was using v1.9.1 with no issues on these executors. If I change the remote executors to a pool with higher memory, then I do not have any issues compiling. Is this intended that rules_kotlin requires more than 1.5GB of memory per action or is this a bug? I provide a unique pool for kotlin actions, but I'd prefer to avoid doing that if this change of increased memory usage is unintentional.

I haven't had time to look into the difference in memory consumed per action. But I did confirm more memory on executor solves the problem. I already have a pool setup for a higher memory amount that passed. The results from minimal testing were:

Error

ERROR: /some/path/BUILD.bazel:24:15: KotlinCompile //:something { kt: 1, java: 0, srcjars: 0 } for k8 failed: (Exit 1): build failed: error executing KotlinCompile command (from target //:something) bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build ... (remaining 1 argument skipped)
Execution result: <redacted>
Exception in thread "main" java.lang.RuntimeException: /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/remotejdk17_linux, [/mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/com_github_jetbrains_kotlin/lib/kotlin-compiler.jar, /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/rules_kotlin/src/main/kotlin/io/bazel/kotlin/compiler/compiler.jar, /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/com_github_jetbrains_kotlin/lib/jvm-abi-gen.jar, /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/rules_kotlin/src/main/kotlin/skip-code-gen.jar, /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/rules_kotlin/src/main/kotlin/jdeps-gen.jar, /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/com_github_google_ksp/symbol-processing-api.jar, /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/com_github_google_ksp/symbol-processing-cmdline.jar, /mnt/worker/work/2/exec/bazel-out/k8-opt-exec-ST-030a29479599/bin/external/rules_kotlin/src/main/kotlin/build.runfiles/com_github_jetbrains_kotlin/lib/kotlin-reflect.jar]
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain.createClassLoader(KotlinToolchain.kt:196)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain.createClassLoader$default(KotlinToolchain.kt:178)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain$classLoader$2.invoke(KotlinToolchain.kt:200)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain$classLoader$2.invoke(KotlinToolchain.kt:199)
    at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain.getClassLoader(KotlinToolchain.kt:199)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain$KotlinCliToolInvoker.<init>(KotlinToolchain.kt:229)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain$K2JSCompilerInvoker.<init>(KotlinToolchain.kt:260)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain_K2JSCompilerInvoker_Factory.newInstance(KotlinToolchain_K2JSCompilerInvoker_Factory.java:39)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain_K2JSCompilerInvoker_Factory.get(KotlinToolchain_K2JSCompilerInvoker_Factory.java:30)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain_K2JSCompilerInvoker_Factory.get(KotlinToolchain_K2JSCompilerInvoker_Factory.java:10)
    at io.bazel.kotlin.builder.dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.bazel.kotlin.builder.tasks.js.Kotlin2JsTaskExecutor_Factory.get(Kotlin2JsTaskExecutor_Factory.java:32)
    at io.bazel.kotlin.builder.tasks.js.Kotlin2JsTaskExecutor_Factory.get(Kotlin2JsTaskExecutor_Factory.java:11)
    at io.bazel.kotlin.builder.dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.bazel.kotlin.builder.tasks.KotlinBuilder_Factory.get(KotlinBuilder_Factory.java:36)
    at io.bazel.kotlin.builder.tasks.KotlinBuilder_Factory.get(KotlinBuilder_Factory.java:12)
    at io.bazel.kotlin.builder.dagger.internal.DoubleCheck.get(DoubleCheck.java:47)
    at io.bazel.kotlin.builder.DaggerKotlinBuilderComponent$KotlinBuilderComponentImpl.work(DaggerKotlinBuilderComponent.java:109)
    at io.bazel.kotlin.builder.cmd.Build$main$1.invoke(Build.kt:35)
    at io.bazel.kotlin.builder.cmd.Build$main$1.invoke(Build.kt:29)
    at io.bazel.worker.Worker$Companion.from(Worker.kt:31)
    at io.bazel.kotlin.builder.cmd.Build.main(Build.kt:29)
Caused by: java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3537)
    at java.base/java.io.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:185)
    at org.jetbrains.kotlin.preloading.ClassPreloadingUtils.loadAllClassesFromJars(ClassPreloadingUtils.java:147)
    at org.jetbrains.kotlin.preloading.ClassPreloadingUtils.preloadClasses(ClassPreloadingUtils.java:49)
    at org.jetbrains.kotlin.preloading.ClassPreloadingUtils.preloadClasses(ClassPreloadingUtils.java:71)
    at io.bazel.kotlin.builder.toolchain.KotlinToolchain.createClassLoader(KotlinToolchain.kt:184)
    ... 22 more

Example

I don't have a separate workspace I can share as this was discovered within my work which uses an internal RBE setup. But this is a simple BUILD file that will fail on our executors with 1.5GB of memory when upgrading to v1.9.2.

load("@rules_kotlin//kotlin:jvm.bzl", "kt_jvm_library")
load("@bazel_skylib//rules:write_file.bzl", "write_file")

write_file(
    name = "write_something",
    out = "Something.kt",
    content = ["""
fun PrintHello() {
    println("Hello, World!")
}
"""],
)

kt_jvm_library(
    name = "something",
    srcs = [":write_something"],
    deps = [],
)
restingbull commented 3 months ago

Unfortunately, you will need to take this up with https://github.com/JetBrains/kotlin

We have not made any revisions to kotlinc.