[GR-52475] Error with native image for ktfmt - Resource not found: /org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class

[cushon]

Describe the issue

I am trying to create a native image for https://github.com/facebook/ktfmt. The native image fails at runtime with a 'Resource not found' error, despite native-image-agent generating a resource-config.json that seems to include the resource.

This looks like the same issue as https://github.com/oracle/graal/issues/4691, which was closed due to inactivity. I am filing a new issue as suggested in https://github.com/oracle/graal/issues/4691#issuecomment-1820739658 since this still reproduces with recent GraalVM versions, and I have included steps to reproduce below.

Steps to reproduce the issue

1. Build https://github.com/facebook/ktfmt

git clone https://github.com/facebook/ktfmt
cd ktfmt
git checkout 494d53bd199db0079d95ed0661b45197ee2d756e
mvn install -DskipTests
...
[INFO] Installing ./core/target/ktfmt-0.47-SNAPSHOT-jar-with-dependencies.jar to ~/.m2/repository/com/facebook/ktfmt/0.47-SNAPSHOT/ktfmt-0.47-SNAPSHOT-jar-with-dependencies.jar

2. Use the agent to generate configs:

echo 'fun f (a : Int) {}' > T.kt
cp ~/.m2/repository/com/facebook/ktfmt/0.47-SNAPSHOT/ktfmt-0.47-SNAPSHOT-jar-with-dependencies.jar ktfmt.jar
$JAVA_HOME/bin/java -agentlib:native-image-agent=config-output-dir=META-INF/native-image -jar ktfmt.kar --dry-run T.kt
zip ktfmt.jar META-INF/native-image/*

3. Build a native image

$JAVA_HOME/bin/native-image --native-image-info --verbose --no-fallback -jar ktfmt.jar

4. The image fails at runtime

$ ./ktfmt T.kt
cannot extract '/org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class' from 'resource:/org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class'
Exception in thread "main" java.lang.ExceptionInInitializerError
    at com.facebook.ktfmt.format.Formatter.sortedAndDistinctImports(Formatter.kt:148)
    at com.facebook.ktfmt.format.Formatter.format(Formatter.kt:94)
    at com.facebook.ktfmt.cli.Main.format(Main.kt:129)
    at com.facebook.ktfmt.cli.Main.access$format(Main.kt:31)
    at com.facebook.ktfmt.cli.Main$run$1.invoke(Main.kt:106)
    at com.facebook.ktfmt.cli.Main$run$1.invoke(Main.kt:104)
    at com.facebook.ktfmt.cli.Main.run$lambda$0(Main.kt:104)
    at java.base@21.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
    at java.base@21.0.2/java.util.Collections$2.tryAdvance(Collections.java:5073)
    at java.base@21.0.2/java.util.Collections$2.forEachRemaining(Collections.java:5081)
    at java.base@21.0.2/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base@21.0.2/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
    at java.base@21.0.2/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:754)
    at java.base@21.0.2/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
    at java.base@21.0.2/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:667)
    at java.base@21.0.2/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
    at java.base@21.0.2/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
    at java.base@21.0.2/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
    at java.base@21.0.2/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596)
    at java.base@21.0.2/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:765)
    at com.facebook.ktfmt.cli.Main.run(Main.kt:104)
    at com.facebook.ktfmt.cli.Main$Companion.main(Main.kt:40)
    at com.facebook.ktfmt.cli.Main.main(Main.kt)
Caused by: java.lang.IllegalStateException: Resource not found: /org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class
    at org.jetbrains.kotlin.utils.PathUtil.getResourcePathForClass(PathUtil.kt:171)
    at org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.registerApplicationExtensionPointsAndExtensionsFrom(KotlinCoreEnvironment.kt:616)
    at org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createApplicationEnvironment(KotlinCoreEnvironment.kt:595)
    at org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.getOrCreateApplicationEnvironment(KotlinCoreEnvironment.kt:505)
    at org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.getOrCreateApplicationEnvironmentForProduction(KotlinCoreEnvironment.kt:492)
    at org.jetbrains.kotlin.cli.jvm.compiler.KotlinCoreEnvironment$Companion.createForProduction(KotlinCoreEnvironment.kt:440)
    at com.facebook.ktfmt.format.Parser.<clinit>(Parser.kt:57)
    ... 23 more

org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class is listed in the generated resource-config.json:

$ grep 'org/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class' META-INF/native-image/resource-config.json
    "pattern":"\\Qorg/jetbrains/kotlin/cli/jvm/compiler/KotlinCoreEnvironment$Companion.class\\E"

I experimented with using --initialize-at-build-time as suggested in https://github.com/oracle/graal/issues/4691#issuecomment-1211745569, but couldn't find an invocation of native-image that succeeded.

Describe GraalVM and your environment:

• GraalVM version (latest snapshot builds can be found here), or commit id if built from source: build 21.0.2+13-LTS-jvmci-23.1-b30
• JDK major version: 21
• OS: macOS Sonoma
• Architecture: aarch64

[nreid260]

I don't think this is an issue with native-images, so much as a particularly bad hack in kotlinc.

From my own experiments, this section tries to dynamically discover from where to load certain kotlinc config files. One of the possible locations is from a classpath JAR of the executing program. To check that case, kotlinc first tries to load the class file of a class known to be in that same JAR (PathUtilKt) as a resource. If that succeeds, it treats the matching JAR as an archive, and loads the config using ZIP file APIs. (This is at least one pathology. There are others using the same PathUtilKt trick.)

See https://github.com/JetBrains/kotlin/blob/master/compiler/util/src/org/jetbrains/kotlin/utils/PathUtil.kt#L168

For native images, this technique can't work. Even if PathUtilKt.class is kept as a resource, there will be not be a JAR in the filesystem.

One of the tip-offs in the error message is the resource: prefix. kotlinc has logic for resolving a few different URI schemes to config file locations. However, resource is not a supported scheme. Native-image conversion seems to lead to suchURIs being created at runtime, when they would not be during execution on a JVM.

See https://github.com/JetBrains/intellij-community/blob/master/platform/util/src/com/intellij/openapi/application/PathManager.java#L461