[GR-53978] Class graph unable to load classes with GraalVM JDK21

[anshu-zs]

Hi I am trying to initialize ClassScanner at build time, but it's not able to load classes

Here is my classScanner:

public class ClassScanner {
    private static ScanResult scanResult;

    static {
        scanResult = new ClassGraph().disableModuleScanning() // added for GraalVM
                 .disableDirScanning() // added for GraalVM
                 .disableNestedJarScanning() // added for GraalVM
                 .disableRuntimeInvisibleAnnotations() // added for GraalVM
                .addClassLoader(ClassLoader.getSystemClassLoader()) // see
                .enableAnnotationInfo().enableMethodInfo().initializeLoadedClasses().scan();
    }

        public Set<Class> findAllClassesImplementingInterfaces(Class interfaceName) {
        Set<Class> classSet = new HashSet<>();
        ClassInfoList classInfos = scanResult.getClassesImplementing(interfaceName);
        for (ClassInfo clazz : classInfos) {
            classSet.add(clazz.loadClass());
        }
        System.out.println("ScanResult:   "+classSet);
        return classSet;
    }

ClassScanner usage

ClassScanner classScanner =new ClassScanner();
classScanner.findAllClassesImplementingInterfaces(IContext.class)

Here are the logs when I run the native image:

INFO [17:48:16]  Default properties successfully read. 
      (Memory: 2621440 Threads: 2)
INFO [17:48:16]  Loaded config from file: .env 
      (Memory: 3145728 Threads: 2)
INFO [17:48:16]  System environment properties successfully read. 
      (Memory: 3145728 Threads: 2)

scanResult: []

INFO [17:48:16]  Initializing dependencies. 
      (Memory: 3145728 Threads: 2)
INFO [17:48:16]  Dependencies successfully created. 
      (Memory: 3145728 Threads: 2)
INFO [17:48:16]  Started Application in 5.042 seconds 
      (Memory: 3145728 Threads: 2)
INFO [17:48:16]  Stopping application 
      (Memory: 4718592 Threads: 4)
INFO [17:48:16]  Application stopped 
      (Memory: 4718592 Threads: 4)

Here as u can see my scanResult shows empty array

---

When I run my application locally instead of running native image I get scanResult as

scanResult: [class com.zopsmart.rocket.context.AppContext, class com.zopsmart.rocket.context.ServerContext]

GraalVM version GraalVM JDK 21

[selhagani]

Hi @anshu-zs,

Thanks for reaching out to us!

I wanted to let you know that there's a compatibility issue between ClassGraph and GraalVM. When compiling to Native Image with GraalVM, the JARs on the classpath are no longer available. This means that any metadata associated with resources in those files becomes invalid.

[anshu-zs]

Hi @anshu-zs,

Thanks for reaching out to us!

I wanted to let you know that there's a compatibility issue between ClassGraph and GraalVM. When compiling to Native Image with GraalVM, the JARs on the classpath are no longer available. This means that any metadata associated with resources in those files becomes invalid.

Hi @selhagani Thank you for replying, So is there any workaround for this?

[selhagani]

Hi @anshu-zs, I understand your concern. Currently, there aren't any workarounds available for this situation. However, I'll take this opportunity to forward your feedback as a feature request to our team. Thank you for bringing this to our attention!

[vjovanov]

One of our latest PRs should help with explaining what to do. We will also write a guide for this kind of cases for the next release.

In this concrete case I would include --initialize-at-build-time=ClassScanner and all follow-up classes that are in the ScanResult. Now note that this will likely make a very large image so you might need to also compute your queries at build time to reduce image size. For example, instead of scanResult you can store the result of findAllClassesImplementingInterfaces to a static field.

[anshu-zs]

Hi @vjovanov I'm already following the guidelines mentioned:

1. I'm already storing the scanResult in static field
2. I am scanning the classes in static block
3. and I have added --initialize-at-build-time=com.zopsmart.rocket.engine.scanner.ClassScanner in command line while creating native image

native-image --no-fallback -H:ConfigurationFileDirectories=/Users/raramuri/Desktop/Anshu/rocket/rocket-examples/src/main/resources/META-INF/native-image    --initialize-at-build-time=org.slf4j.LoggerFactory,ch.qos.logback,io.netty,examples.http.controller.ProductController,io.github.classgraph,nonapi.io.github.classgraph,examples.http.Main,com.zopsmart.rocket.engine.scanner.ClassScanner    -H:+ReportUnsupportedElementsAtRuntime  -H:+ReportExceptionStackTraces --report-unsupported-elements-at-runtime    -H:+AllowJRTFileSystem  -H:Class=examples.http.Main  -jar rocket-examples-0.31.0.jar

---

Here is modified ClassScanner class

    static {
        scanResult =
                new ClassGraph()
                        .enableAnnotationInfo()
                        .enableMethodInfo()
                        .enableFieldInfo()
                        .scan();

        System.out.println("Scanner:" + scanResult.getClassesImplementing(IEngine.class));
    }

I am basically trying to print the result of scanResult.getClassesImplementing(IEngine.class)

---

For version GraalVM CE 22.3.1, 22.3.3 I'm getting this output while building native image

========================================================================================================================
GraalVM Native Image: Generating 'rocket-examples-0.31.0' (executable)...
========================================================================================================================
Scanner:[@lombok.Generated public class com.zopsmart.rocket.engine.ApiMappingEngine implements public abstract com.zopsmart.rocket.engine.IEngine, public class com.zopsmart.rocket.engine.BeanFactoryEngine implements public abstract com.zopsmart.rocket.engine.IEngine, public class com.zopsmart.rocket.engine.ConfigEngine implements public abstract com.zopsmart.rocket.engine.IEngine, public class com.zopsmart.rocket.engine.DependencyEngine implements public abstract com.zopsmart.rocket.engine.IEngine, public class com.zopsmart.rocket.engine.HealthEngine implements public abstract com.zopsmart.rocket.engine.IEngine, public class com.zopsmart.rocket.engine.KeyEngine implements public abstract com.zopsmart.rocket.engine.IEngine, public class com.zopsmart.rocket.engine.MetricsEngine implements public abstract com.zopsmart.rocket.engine.IEngine, @lombok.Generated public class com.zopsmart.rocket.engine.ServerEngine implements public abstract com.zopsmart.rocket.engine.IEngine, @lombok.Generated public class com.zopsmart.rocket.engine.TracerEngine implements public abstract com.zopsmart.rocket.engine.IEngine]

Warning: Could not register io.netty.handler.codec.marshalling.MarshallingEncoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: org/jboss/marshalling/ByteOutput.
Warning: Could not register io.netty.handler.codec.compression.Lz4FrameDecoder: queryAllPublicMethods for reflection. Reason: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Exception.
.....

PS: Observe the result stored in Scanner but eventually I end up with this error -> mentioned in https://github.com/oracle/graal/issues/8883

Error: java.util.concurrent.ExecutionException: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: type is not available in this platform: com.oracle.svm.hosted.NativeImageSystemClassLoader

---

For other versions (eg: graalvm-jdk-21.0.2+13.1) My scanner result is empty

========================================================================================================================
GraalVM Native Image: Generating 'rocket-examples-0.31.0' (executable)...
========================================================================================================================
Scanner:[]     

[vjovanov]

The first error comes from the fact that the scanner pulled in native image classes. In later versions on GraalVM this should not happen.

In the second snippet, the issue is that the used class loader does not see the classes from the application. Can you try to pass the ClassScanner.class.getClassLoader() as the class loader to the ClassGraph.

[selhagani]

I noticed that since upgrading to GraalVM for JDK 22, you've encountered a new error. The error message suggests using the 'onlyWith' field in the TargetClass annotation to make substitution active only when necessary. It might be beneficial to review how your substitution-related annotations are configured. To provide more specific assistance, I really need a concise reproducer for the issue. Thanks!

[vjovanov]

@selhagani for which example? Can you post an error?

[anshu-zs]

The first error comes from the fact that the scanner pulled in native image classes. In later versions on GraalVM this should not happen.

In the second snippet, the issue is that the used class loader does not see the classes from the application. Can you try to pass the ClassScanner.class.getClassLoader() as the class loader to the ClassGraph.

Hi @vjovanov I tried adding classLoader to classgraph

public class ClassScanner {
    private static ScanResult scanResult;

    static {
        scanResult =
                new ClassGraph()
                        .addClassLoader(ClassScanner.class.getClassLoader())
                        .enableAnnotationInfo()
                        .enableMethodInfo().enableFieldInfo().scan();
        System.out.println("Scanner:" + scanResult.getClassesImplementing(IEngine.class));
    }

        public Set<Class> findAllClassesImplementingInterfaces(Class interfaceName) {
        Set<Class> classSet = new HashSet<>();
        ClassInfoList classInfos = scanResult.getClassesImplementing(interfaceName);
        for (ClassInfo clazz : classInfos) {
            classSet.add(clazz.loadClass());
        }
        System.out.println("ScanResult:   "+classSet);
        return classSet;
    }
}

and tried creating the native image using latest Graal VM: GraalVM CE 22.0.1+8.1, but still I got the Scanner as empty along with the error

Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.grpc.netty.shaded.io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:IncludeResources=application\.properties' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:IncludeResources=application\.conf' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:IncludeResources=application\.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:IncludeResources=.*Driver\.properties' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:DynamicProxyConfigurationResources=META-INF/native-image/com.datastax.oss/java-driver-core/proxy.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:+ReportUnsupportedElementsAtRuntime' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/com.datastax.oss/java-driver-core/reflection.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:IncludeResources=reference\.conf' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:+AllowJRTFileSystem' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: The option '-H:ReflectionConfigurationResources=META-INF/native-image/io.netty/netty-transport/reflection-config.json' is experimental and must be enabled via '-H:+UnlockExperimentalVMOptions' in the future.
Warning: Please re-evaluate whether any experimental option is required, and either remove or unlock it. The build output lists all active experimental options, including where they come from and possible alternatives. If you think an experimental option should be considered as stable, please file an issue.
========================================================================================================================
GraalVM Native Image: Generating 'rocket-examples-0.31.0' (executable)...
========================================================================================================================
Scanner:[]

[1/8] Initializing...                                                                                    (0.0s @ 0.27GB)
Error: Substitution target for org.wildfly.common.Substitutions$Target_GraalDirectives is not loaded. Use field `onlyWith` in the `TargetClass` annotation to make substitution only active when needed.
------------------------------------------------------------------------------------------------------------------------
                        1.0s (21.1% of total time) in 45 GCs | Peak RSS: 0.97GB | CPU load: 4.27
========================================================================================================================
Failed generating 'rocket-examples-0.31.0' after 4.1s.

[selhagani]

Hey @vjovanov, apologies for the confusion! It seems I responded to the wrong issue that @anshu-zs has also shared and mentioned this issue in. My comment was meant for that issue not this one. Thanks for your understanding!

I noticed that since upgrading to GraalVM for JDK 22, you've encountered a new error. The error message suggests using the 'onlyWith' field in the TargetClass annotation to make substitution active only when necessary. It might be beneficial to review how your substitution-related annotations are configured. To provide more specific assistance, I really need a concise reproducer for the issue. Thanks!

[vjovanov]

I looked into this and it seems that ClassGraph does not see the classpath when it is scanning, however it can load all the classes with the correct class loader. The only solution I found was to add the following line:

   new ClassGraph().overrideClasspath("<image-builder-classpath>")

and the results showed up.

@olpaw do we have a way to fetch image-builder-classpath from the static initializer? Also, do you know what is the reason that we deleted java.class.path at build time?

@anshu-zs I would try to ask a question on the classgraph issue tracker or mailing list to see why it does not see the path of the NativeImageClassLoader.

[anshu-zs]

sure @vjovanov , thanks for the help. please do let me know about the classgraph issue whenever you find anything