Remove 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' for Spring Boot 3.2+ Applications

[KafkaProServerless]

Hello team,

I just downloaded the latest 3.3.0-RC1, and wanted to build a native image out of it, with GraalVM 22.

With the simple maven command mvn -Pnative spring-boot:build-image with this pom

<plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <env>
                            <BP_JVM_VERSION>22</BP_JVM_VERSION>
                        </env>
                    </image>
                </configuration>
            </plugin>

I am able to reproduce this 100%:

[INFO]     [creator]     [2/8] Performing analysis...  []                                (30.6s @ 0.82GB)
[INFO]     [creator]        10,667 reachable types   (74.8% of   14,256 total)
[INFO]     [creator]        14,669 reachable fields  (50.2% of   29,204 total)
[INFO]     [creator]        43,373 reachable methods (46.3% of   93,752 total)
[INFO]     [creator]         4,399 types,   848 fields, and 6,558 methods registered for reflection
[INFO]     [creator]     
[INFO]     [creator]     Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: An object of type 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.
[INFO]     [creator]     This is not allowed for correctness reasons: All objects that are stored in the image heap must be initialized at build time.
[INFO]     [creator]     
[INFO]     [creator]     You now have two options to resolve this:
[INFO]     [creator]     
[INFO]     [creator]     1) If it is intended that objects of type 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' are persisted in the image heap, add 
[INFO]     [creator]     
[INFO]     [creator]         '--initialize-at-build-time=org.springframework.boot.loader.nio.file.NestedFileSystemProvider'
[INFO]     [creator]     
[INFO]     [creator]     to the native-image arguments. Note that initializing new types can store additional objects to the heap. It is advised to check the static fields of 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' to see if they are safe for build-time initialization,  and that they do not contain any sensitive data that should not become part of the image.
[INFO]     [creator]     
[INFO]     [creator]     2) If these objects should not be stored in the image heap, you can use 
[INFO]     [creator]     
[INFO]     [creator]         '--trace-object-instantiation=org.springframework.boot.loader.nio.file.NestedFileSystemProvider'
[INFO]     [creator]     
[INFO]     [creator]     to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 
[INFO]     [creator]     
[INFO]     [creator]         '--initialize-at-run-time=<culprit>'
[INFO]     [creator]     
[INFO]     [creator]     to prevent the instantiation of the object.
[INFO]     [creator]     
[INFO]     [creator]     If you are seeing this message after upgrading to a new GraalVM release, this means that some objects ended up in the image heap without their type being marked with --initialize-at-build-time.
[INFO]     [creator]     To fix this, include '--initialize-at-build-time=org.springframework.boot.loader.nio.file.NestedFileSystemProvider' in your configuration. If the classes do not originate from your code, it is advised to update all library or framework dependencies to the latest version before addressing this error.
[INFO]     [creator]     
[INFO]     [creator]     The following detailed trace displays from which field in the code the object was reached.
[INFO]     [creator]     Object was reached by
[INFO]     [creator]       manually created constant
[INFO]     [creator]     Error: java.util.concurrent.ExecutionException: com.oracle.graal.pointsto.util.AnalysisError: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: An object of type 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.
[INFO]     [creator]     This is not allowed for correctness reasons: All objects that are stored in the image heap must be initialized at build time.
[INFO]     [creator]     
[INFO]     [creator]     You now have two options to resolve this:
[INFO]     [creator]     
[INFO]     [creator]     1) If it is intended that objects of type 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' are persisted in the image heap, add 
[INFO]     [creator]     
[INFO]     [creator]         '--initialize-at-build-time=org.springframework.boot.loader.nio.file.NestedFileSystemProvider'
[INFO]     [creator]     
[INFO]     [creator]     to the native-image arguments. Note that initializing new types can store additional objects to the heap. It is advised to check the static fields of 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' to see if they are safe for build-time initialization,  and that they do not contain any sensitive data that should not become part of the image.
[INFO]     [creator]     
[INFO]     [creator]     2) If these objects should not be stored in the image heap, you can use 
[INFO]     [creator]     
[INFO]     [creator]         '--trace-object-instantiation=org.springframework.boot.loader.nio.file.NestedFileSystemProvider'
[INFO]     [creator]     
[INFO]     [creator]     to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 
[INFO]     [creator]     
[INFO]     [creator]         '--initialize-at-run-time=<culprit>'
[INFO]     [creator]     
[INFO]     [creator]     to prevent the instantiation of the object.
[INFO]     [creator]     
[INFO]     [creator]     If you are seeing this message after upgrading to a new GraalVM release, this means that some objects ended up in the image heap without their type being marked with --initialize-at-build-time.
[INFO]     [creator]     To fix this, include '--initialize-at-build-time=org.springframework.boot.loader.nio.file.NestedFileSystemProvider' in your configuration. If the classes do not originate from your code, it is advised to update all library or framework dependencies to the latest version before addressing this error.
[INFO]     [creator]     
[INFO]     [creator]     The following detailed trace displays from which field in the code the object was reached.
[INFO]     [creator]     Object was reached by
[INFO]     [creator]       manually created constant
[INFO]     [creator]     --------------------------------------------------------------------------------
[INFO]     [creator]         3.6s (9.4% of total time) in 73 GCs | Peak RSS: 1.99GB | CPU load: 6.20
[INFO]     [creator]     ================================================================================
[INFO]     [creator]     Failed generating 'com.Application' after 37.7s.
[INFO]     [creator]     unable to invoke layer creator
[INFO]     [creator]     unable to contribute native-image layer
[INFO]     [creator]     error running build
[INFO]     [creator]     exit status 1
[INFO]     [creator]     ERROR: failed to build: exit status 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

Could you please help on this UnsupportedFeatureException issue?

[KafkaProServerless]

https://github.com/spring-projects/spring-boot/issues/40570

[dmikusa]

When there is an issue with the native-image compiler, this is very infrequently a buildpacks problem. The native-image buildpack is just running the compiler with some default options, and most failures like this are due to the app being built.

I see on the Spring issues that it works when building locally, but it doesn't indicate that the same arguments were used locally so I would suggest that you take a look at the options that work when you build locally and see if you can set BP_NATIVE_IMAGE_BUILD_ARGUMENTS to get your app to build with the same set of args. The buildpack will supply a default set of options, but you can adjust those as necessary to make your app build correctly.

In some rare cases, the buildpack will not allow you to override options. If one of these non-overridable options is causing an issue, please let me know and we can talk about a fix for that.

[KafkaProServerless]

issue.zip

[KafkaProServerless]

Here is a minimal reproducible sample

[KafkaProServerless]

just need to add :

<configuration>
  <image>
    <env>
      <BP_JVM_VERSION>22</BP_JVM_VERSION>
    </env>
  </image>
</configuration>

[KafkaProServerless]

Hello @dmikusa , thank you for your answer.

On my side, I have no other build other than the native image build.

I also do not know how to "dump" build arguments. I did not add any on my side.

I have attached a raw example, minimal.

By just adding the configuration on the springboot maven plugin, it is 100% reproducible.

May I ask if you could help download, add the configuration, run "mvn -Pnative spring-boot:build-image" and let me know if you reproduce the same please?

[KafkaProServerless]

Hello @dmikusa , could you please help?

[scottfrederick]

The root cause of this problem is https://github.com/oracle/graal/issues/5134.

Starting with Spring Boot 3.2, the Spring Boot loader that is packaged in a Boot jar file includes the org.springframework.boot.loader.nio.file.NestedFileSystemProvider class and a META-INF/services/java.nio.file.spi.FileSystemProvider file that references the NestedFileSystemProvider and causes it to be loaded in the JVM. This is triggering the GraalVM issue.

When building a native image locally using ./mvnw native:compile -Pnative, the loader code is not included in the classpath sent to the native-image command, so the image build succeeds.

You can work around this problem by adding this configuration to the pom.xml file:

      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <configuration>
          <image>
            <env>
              <BP_NATIVE_IMAGE_BUILD_ARGUMENTS>-H:-AddAllFileSystemProviders</BP_NATIVE_IMAGE_BUILD_ARGUMENTS>
            </env>
          </image>
        </configuration>
      </plugin>

The loader code is not needed when building a native image from a Spring Boot application, so it could be removed. We'll have to think more about the best way to address this long-term.

We could consider excluding the loader from the content that is sent to the buildpacks with ./mvnw -Pnative spring-boot:build-image, but the same error happens if you build from source with pack using pack build --builder paketobuildpacks/builder-jammy-base:latest --path . issue --env BP_NATIVE_IMAGE=true --env BP_JVM_VERSION=22 (without the Spring Boot CNB integration involved), so this might be something that buildpacks would need to do.

[KafkaProServerless]

Thank you

[dmikusa]

@scottfrederick We can do something to make sure this is not included at the buildpack level, so this isn't triggered. I guess my question would be what to remove?

It seems like if META-INF/services/java.nio.file.spi.FileSystemProvider exists and then we could remove it. If that is what pulls the file in when native-image runs, then removing it would seem to fix the issue. I'm not sure if that is overly aggressive though, like if that file could be used for other things where it would need to exist to build properly.

Do you think that would be ok @scottfrederick ?

[philwebb]

Perhaps removing org.springframework.boot.loader.nio.file.NestedFileSystemProvider from the contents of the file might work and only deleting it if that's the only line. That might be a bit safer.

[dmikusa]

We can do that. Thanks @philwebb

[anthonydahanne]

currently working on this, but from the https://github.com/paketo-buildpacks/spring-boot code base; since this issue is specific to Spring Boot, not native-image

[anthonydahanne]

it was fixed in https://github.com/paketo-buildpacks/spring-boot/issues/488 - the next release of Spring Boot buildpack will have the fix (end of the week)

[KafkaProServerless]

Thank you, will test when 3.3.1 launches

[KafkaProServerless]

Hello team, hello @anthonydahanne ,

Thank you for the work on this. On my side, I updated to 3.3.1, but still encountering the issue.

Now with 3.3.1, the pom looks like:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.3.1</version>
        <relativePath/>
    </parent>

   <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <image>
                        <env>
                            <BP_JVM_VERSION>22</BP_JVM_VERSION>

                        </env>
                    </image>
                </configuration>
            </plugin>

But this still gives error:

[INFO]     [creator]     [1/8] Initializing...                                            (7.3s @ 0.34GB)
[INFO]     [creator]      Java version: 22.0.1+10, vendor version: Liberica-NIK-24.0.1-1
[INFO]     [creator]      Graal compiler: optimization level: 2, target machine: x86-64-v3
[INFO]     [creator]      C compiler: gcc (linux, x86_64, 11.4.0)
[INFO]     [creator]      Garbage collector: Serial GC (max heap size: 80% of RAM)
[INFO]     [creator]      2 user-specific feature(s):
[INFO]     [creator]      - com.oracle.svm.thirdparty.gson.GsonFeature
[INFO]     [creator]      - org.springframework.aot.nativex.feature.PreComputeFieldFeature
[INFO]     [creator]     --------------------------------------------------------------------------------
[INFO]     [creator]      2 experimental option(s) unlocked:
[INFO]     [creator]      - '-H:Name' (alternative API option(s): -o com.MyApplication; origin(s): command line)
[INFO]     [creator]      - '-H:+StaticExecutableWithDynamicLibC' (origin(s): command line)
[INFO]     [creator]     --------------------------------------------------------------------------------
[INFO]     [creator]     Build resources:
[INFO]     [creator]      - 18.82GB of memory (60.4% of 31.18GB system memory, determined at start)
[INFO]     [creator]      - 8 thread(s) (100.0% of 8 available processor(s), determined at start)
[INFO]     [creator]     SLF4J(W): No SLF4J providers were found.
[INFO]     [creator]     SLF4J(W): Defaulting to no-operation (NOP) logger implementation
[INFO]     [creator]     SLF4J(W): See https://www.slf4j.org/codes.html#noProviders for further details.
[INFO]     [creator]     [2/8] Performing analysis...  []                                (31.6s @ 1.41GB)
[INFO]     [creator]        13,957 reachable types   (78.9% of   17,691 total)
[INFO]     [creator]        19,776 reachable fields  (50.9% of   38,828 total)
[INFO]     [creator]        58,779 reachable methods (50.9% of  115,548 total)
[INFO]     [creator]         5,224 types, 1,067 fields, and 7,549 methods registered for reflection
[INFO]     [creator]     
[INFO]     [creator]     Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: An object of type 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' was found in the image heap. This type, however, is marked for initialization at image run time for the following reason: classes are initialized at run time by default.
[INFO]     [creator]     This is not allowed for correctness reasons: All objects that are stored in the image heap must be initialized at build time.
[INFO]     [creator]     
[INFO]     [creator]     You now have two options to resolve this:
[INFO]     [creator]     
[INFO]     [creator]     1) If it is intended that objects of type 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' are persisted in the image heap, add 
[INFO]     [creator]     
[INFO]     [creator]         '--initialize-at-build-time=org.springframework.boot.loader.nio.file.NestedFileSystemProvider'
[INFO]     [creator]     
[INFO]     [creator]     to the native-image arguments. Note that initializing new types can store additional objects to the heap. It is advised to check the static fields of 'org.springframework.boot.loader.nio.file.NestedFileSystemProvider' to see if they are safe for build-time initialization,  and that they do not contain any sensitive data that should not become part of the image.
[INFO]     [creator]     
[INFO]     [creator]     2) If these objects should not be stored in the image heap, you can use 
[INFO]     [creator]     
[INFO]     [creator]         '--trace-object-instantiation=org.springframework.boot.loader.nio.file.NestedFileSystemProvider'
[INFO]     [creator]     
[INFO]     [creator]     to find classes that instantiate these objects. Once you found such a class, you can mark it explicitly for run time initialization with 
[INFO]     [creator]     
[INFO]     [creator]         '--initialize-at-run-time=<culprit>'
[INFO]     [creator]     
[INFO]     [creator]     to prevent the instantiation of the object.
[INFO]     [creator]     
[INFO]     [creator]     If you are seeing this message after upgrading to a new GraalVM release, this means that some objects ended up in the image heap without their type being marked with --initialize-at-build-time.
[INFO]     [creator]     To fix this, include '--initialize-at-build-time=org.springframework.boot.loader.nio.file.NestedFileSystemProvider' in your configuration. If the classes do not originate from your code, it is advised to update all library or framework dependencies to the latest version before addressing this error.
[INFO]     [creator]     
[INFO]     [creator]     The following detailed trace displays from which field in the code the object was reached.
[INFO]     [creator]     Object was reached by
[INFO]     [creator]       manually created constant
[INFO]     [creator]     --------------------------------------------------------------------------------
[INFO]     [creator]         3.1s (7.7% of total time) in 51 GCs | Peak RSS: 2.59GB | CPU load: 6.76
[INFO]     [creator]     ================================================================================
[INFO]     [creator]     Failed generating 'com.MyApplication' after 39.8s.
[INFO]     [creator]     unable to invoke layer creator
[INFO]     [creator]     unable to contribute native-image layer
[INFO]     [creator]     error running build
[INFO]     [creator]     exit status 1
[INFO]     [creator]     ERROR: failed to build: exit status 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------

If I add back the <BP_NATIVE_IMAGE_BUILD_ARGUMENTS>-H:-AddAllFileSystemProviders</BP_NATIVE_IMAGE_BUILD_ARGUMENTS> le build passes.

I would have expected with this fix, that I would not need to pass in <BP_NATIVE_IMAGE_BUILD_ARGUMENTS>-H:-AddAllFileSystemProviders</BP_NATIVE_IMAGE_BUILD_ARGUMENTS>

May I ask what did I do wrong?

Thank you