Providers loaded by FileSystemProvider not handled correctly in native image

[OlgaMaciaszek]

Providers loaded by java.nio.file.spi.FileSystemProvider (such as BouncyCastleProvider or SftpFileSystemProvider) cause native image compilation to fail with No instances of xxx are allowed in the image heap as this class should be initialized at image runtime.. However, adding --initialize-at-run-time=java.nio.file.spi.FileSystemProvider to setup results in Error: The class java.nio.file.spi.FileSystemProvider has already been initialized (from command line with 'java.nio.file.spi.FileSystemProvider'); it is too late to register java.nio.file.spi.FileSystemProvider for build-time initialization..

You can use this demo branch as a sample. Run with ./mvnw clean package -Pnative -DskipTests.

[wirthi]

@vjovanov will look into that, but can't promise we can resolve this by the end of the week (for GraalVM-22.3).

[Dam14n]

Hi guys, I'm facing the same issue. Do you have any news on this?

Can I help with something?

[vjovanov]

This requires moving FileSystemProviders to runtime. I tried making this change and many things failed because of the initialization strategy we use. I would wait for the new initialization strategy before addressing this issue.

[OlgaMaciaszek]

Thanks, @vjovanov. What would the timeline be for this?

[mikehearn]

I'm hitting this too. It's annoying because the FS provider is brought in via transitive dependencies.

[bkesmarki]

I'd also be interested in any news about the issue. We are using a custom EnvironmentRepository, but from the result/symptoms perspective, we are blocked by the same error with graalvm, native image execution.

[gopalharis]

We are in need of the fix for this issue, this is the only blocker we have for graalvm native build

[sschuberth]

such as BouncyCastleProvider or SftpFileSystemProvider

FYI, in case of SftpFileSystemProvider from Apache MINA, coming in as a transitive dependency of JGit, I was able to work around the issue by simply excluding the dependency, because SFTP is usually not needed as a Git transport:

implementation("org.eclipse.jgit:org.eclipse.jgit.ssh.apache.agent:6.7.0.202309050840-r") {
    exclude(group = "org.apache.sshd", module = "sshd-sftp")
        .because("it is not required for cloning via SSH and causes issues with GraalVM native images")
}

[avollmaier]

I also need a fix for this bug - blocker

[dudiao]

When will it be repaired?

[vjovanov]

Would setting up the -H:-AddAllFileSystemProviders at image build help as a temporary workaround?

[OlgaMaciaszek]

@vjovanov - have tried that, but got a different issue then:

Fatal error: com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing org.apache.sshd.common.config.keys.FilePasswordProvider.<clinit>(FilePasswordProvider.java:55) 
Parsing context:
   at static root method.(Unknown Source)

        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:149)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:178)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:152)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraphInfo(MethodTypeFlow.java:110)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultAnalysisPolicy.staticRootMethodGraph(DefaultAnalysisPolicy.java:199)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.PointsToAnalysis.lambda$addRootMethod$0(PointsToAnalysis.java:315)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.PointsToAnalysis$2.run(PointsToAnalysis.java:493)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1655)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1622)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:165)
Caused by: org.graalvm.compiler.java.BytecodeParser$BytecodeParserError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of org.apache.sshd.common.config.keys.FilePasswordProvider$$Lambda$357/0x0000000800d5a168 are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=org.apache.sshd.common.config.keys.FilePasswordProvider$$Lambda$357/0x0000000800d5a168.
        at parsing org.apache.sshd.common.config.keys.FilePasswordProvider.<clinit>(FilePasswordProvider.java:55)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.throwParserError(BytecodeParser.java:2536)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.throwParserError(SharedGraphBuilderPhase.java:169)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3414)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.iterateBytecodesForBlock(SharedGraphBuilderPhase.java:712)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.handleBytecodeBlock(BytecodeParser.java:3366)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3208)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1134)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.build(SharedGraphBuilderPhase.java:152)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:1026)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:97)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:114)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:434)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.AnalysisParsedGraph.parseBytecode(AnalysisParsedGraph.java:146)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.parseGraph(AnalysisMethod.java:819)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsedHelper(AnalysisMethod.java:784)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsed(AnalysisMethod.java:767)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:184)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:583)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:165)
        ... 13 more
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: No instances of org.apache.sshd.common.config.keys.FilePasswordProvider$$Lambda$357/0x0000000800d5a168 are allowed in the image heap as this class should be initialized at image runtime. To see how this object got instantiated use --trace-object-instantiation=org.apache.sshd.common.config.keys.FilePasswordProvider$$Lambda$357/0x0000000800d5a168.
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.classinitialization.ClassInitializationFeature.checkImageHeapInstance(ClassInitializationFeature.java:132)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisUniverse.replaceObject(AnalysisUniverse.java:616)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin$Transplanter.constant(IntrinsifyMethodHandlesInvocationPlugin.java:970)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin$Transplanter.node(IntrinsifyMethodHandlesInvocationPlugin.java:851)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin$Transplanter.graph(IntrinsifyMethodHandlesInvocationPlugin.java:705)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin.processInvokeWithMethodHandle(IntrinsifyMethodHandlesInvocationPlugin.java:639)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.IntrinsifyMethodHandlesInvocationPlugin.handleInvoke(IntrinsifyMethodHandlesInvocationPlugin.java:263)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.tryNodePluginForInvocation(BytecodeParser.java:2312)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.appendInvoke(BytecodeParser.java:1887)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genDynamicInvokeHelper(BytecodeParser.java:1764)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genInvokeDynamic(BytecodeParser.java:1715)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genInvokeDynamic(BytecodeParser.java:1711)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.AnalysisGraphBuilderPhase$AnalysisBytecodeParser.genInvokeDynamic(AnalysisGraphBuilderPhase.java:139)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5333)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3406)
        ... 32 more

Is there a way to specify FileSystemProviders to add?

[vjovanov]

This looks like a message from older versions of GraalVM. Have you tried with the latest dev build, or with the GraalVM for JDK 21 with a flag --strict-image-heap?

[OlgaMaciaszek]

That was from 17.0.9. Now from 21.0.1, getting Error: Classes that should be initialized at run time got initialized during image building followed by a lot of classes, big part of them BounceCastle-related. Adding --initialize-at-build-time options has fixed it. However, there's now another issue: during initialization of BouncyCastle classes, we get issues related to SP800SecureRandom being initialized at build time, but then if we set --initialize-at-run-time=org.bouncycastle.jcajce.provider.drbg.DRBG (that being the class the instantiation can be traced to), then we get:

 The class org.bouncycastle.jcajce.provider.drbg.DRBG has already been initialized (from '[property source]' with 'org.bouncycastle.jcajce.provider.drbg.DRBG'); it is too late to register org.bouncycastle.jcajce.provider.drbg.DRBG for build-time initialization.

[vjovanov]

I think we need to add the option --strict-image-heap to make it work. I could not use the reproducer as it uses Spring snapshots. What is the right combination of versions for Java 21?

[OlgaMaciaszek]

That would be 6.1.x snapshots; but I'll update the sample build config with snapshot repos today and let you know - sorry for missing that. Also, I have managed to make a native SC Config server work with this workaround as a native image for some basic scenarios. Testing some more scenarios.

[OlgaMaciaszek]

So, actually, after switching to this setup and fixing java.security providers config (I think there's an issue in the doc, have created a PR with fix suggestion, waiting for approval to accept CLI: https://github.com/oracle/graal/pull/7855), I was able to run it with -H:-AddAllFileSystemProviders, without the exclusion. I guess a reproducer is not exactly needed at this point, but I'm adding updated info for documentation purposes: SC Config branch of SC Config (./mvnw clean install -Pspring to build), then use it to create and run a native image on this demo branch (./mvnw native:compile -Pnative and ./target/demo-config-server-ref) . Thanks for your help @vjovanov .

[vjovanov]

Hope this can be a workaround for everyone until this issue is fixed as a part of https://github.com/oracle/graal/issues/7488.

[OlgaMaciaszek]

It does work, however, since it requires initialising logback-related classes at build time to work, and after chatting with the team, the decision is we'll not be able to support it, so still a blocker. I will take another look at the exclusion workaround and post here.

[vjovanov]

Have you tried to reduce that list of initialized classes when the --strict-image-heap is used?

[OlgaMaciaszek]

Thanks for the suggestion @vjovanov. With --strict-image-heap and -H:-AddAllFileSystemProviders (without the exclude), I was able to make a native image Config server with git backend over ssh work with only EdDSASecurityProvider and the org.bouncycastle` components initialised at build time (setup here) - I think there's no way around it since any security providers need to be initialised then anyhow? Will be testing some other scenarios now.

[vjovanov]

We will try to move security classes to run-time init in the future, but it is uncertain whether it will be possible. Until then this looks like a good solution.

[KafkaProServerless]

Impacted by this

[andreiyusupau]

@OlgaMaciaszek maybe a stupid question, but where exactly args like -H:-AddAllFileSystemProvider should be added? I use gradle with following config, but it doesn't really seems to help:

graalvmNative {
    binaries {
        main {
            buildArgs.add('-H:-AddAllFileSystemProviders')
            buildArgs.add('--strict-image-heap')
            }
            }
            }

[OlgaMaciaszek]

@andreiyusupau It does work correctly in our sample setup: https://github.com/spring-projects/spring-aot-smoke-tests/blob/main/cloud/cloud-config-server/build.gradle. You need to make sure you have the right version of Graal (21 or higher).