search

oracle / graal

GraalVM compiles Java applications into native executables that start instantly, scale fast, and use fewer compute resources 🚀
https://www.graalvm.org
Other
20.34k stars 1.63k forks source link

`org.reflections8.Reflections.getTypesAnnotatedWith()` don't work after native-images #3075

Open cyw3 opened 3 years ago

cyw3 commented 3 years ago

Describe the issue I use org.reflections8.Reflections.getTypesAnnotatedWith() to get class and do somethings. When I run this java code, org.reflections8.Reflections.getTypesAnnotatedWith() return a list with lots of class.

But after native-images, org.reflections8.Reflections.getTypesAnnotatedWith() return a empty list.

Steps to reproduce the issue

  1. Here is demo:
public class Main {
    public static void main(String[] args) {
        Reflections ref = new Reflections("xxx");
        for (Class<?> clz : ref.getTypesAnnotatedWith(VisitorAnnotation.class)) {
            System.out.println(clz);
        }
    }
}

@VisitorAnnotation
public class GoFuncVisitor {
}
  1. after native-images with the cmd: 
    
> $GRAALVM_HOME/bin/native-image -jar /Users/xxx/debug.jar -H:Name=Debug -H:+ReportUnsupportedElementsAtRuntime -H:+ReportExceptionStackTraces -H:ReflectionConfigurationFiles=reflect-config.json -R:-RemoveNeverExecutedCode -H:-RemoveNeverExecutedCode

[Debug:67713] classlist: 5,655.88 ms, 2.15 GB [Debug:67713] (cap): 1,181.31 ms, 2.15 GB [Debug:67713] setup: 2,607.21 ms, 2.23 GB [Debug:67713] (clinit): 362.57 ms, 2.48 GB [Debug:67713] (typeflow): 8,199.46 ms, 2.48 GB [Debug:67713] (objects): 9,683.31 ms, 2.48 GB [Debug:67713] (features): 1,071.99 ms, 2.48 GB [Debug:67713] analysis: 19,700.38 ms, 2.48 GB [Debug:67713] universe: 798.99 ms, 2.61 GB [Debug:67713] (parse): 1,545.43 ms, 2.61 GB [Debug:67713] (inline): 1,695.35 ms, 2.62 GB [Debug:67713] (compile): 14,076.83 ms, 3.22 GB [Debug:67713] compile: 18,250.54 ms, 3.22 GB [Debug:67713] image: 1,744.64 ms, 3.22 GB [Debug:67713] write: 745.69 ms, 3.22 GB [Debug:67713] [total]: 49,705.18 ms, 3.22 GB

`org.reflections8.Reflections.getTypesAnnotatedWith()` return a empty list.

Here is reflect-config.json:
```json
[
  {
    "name": "xxx.Scanner",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allDeclaredClasses": true,
    "allPublicClasses": true,
    "allDeclaredFields": true,
    "allPublicFields": true
  },
  {
    "name": "xxx.VisitorAnnotation",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allDeclaredClasses": true,
    "allPublicClasses": true,
    "allDeclaredFields": true,
    "allPublicFields": true
  },
  {
    "name": "org.antlr.v4.runtime.ParserRuleContext",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allDeclaredClasses": true,
    "allPublicClasses": true,
    "allDeclaredFields": true,
    "allPublicFields": true
  },
  {
    "name": "java.util.List",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allDeclaredClasses": true,
    "allPublicClasses": true,
    "allDeclaredFields": true,
    "allPublicFields": true
  },
  {
    "name": "java.util.Iterator",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allDeclaredClasses": true,
    "allPublicClasses": true,
    "allDeclaredFields": true,
    "allPublicFields": true
  },
  {
    "name": "java.util.ArrayList",
    "allDeclaredConstructors": true,
    "allPublicConstructors": true,
    "allDeclaredMethods": true,
    "allPublicMethods": true,
    "allDeclaredClasses": true,
    "allPublicClasses": true,
    "allDeclaredFields": true,
    "allPublicFields": true
  }
]

I don't know How many classes relate to Reflection.

Describe GraalVM and your environment:

munishchouhan commented 3 years ago

@cyw3 use native-image tracing agent to create reflect-config.json https://medium.com/graalvm/introducing-the-tracing-agent-simplifying-graalvm-native-image-configuration-c3b56c486271

cyw3 commented 3 years ago

OK, I try native-image-agent now.

But I encounter some problems:

  1. the same error as #2998. And after I change to use graalvm-jdk11, it works.

  2. org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]

Steps to reproduce the issue

java -agentlib:native-image-agent=config-output-dir=META-INF/native-image -jar debug.jar -m
ant clean jar
native-image -jar debug.jar -H:Name=debug -H:+ReportUnsupportedElementsAtRuntime -H:+ReportExceptionStackTraces -R:-RemoveNeverExecutedCode -H:-RemoveNeverExecutedCode -H:+AllowIncompleteClasspath
./debug -m
[main] WARN org.reflections8.Reflections - could not create Vfs.Dir from url. ignoring the exception and continuing
org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:118)
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:100)
    at org.reflections8.Reflections.scan(Reflections.java:278)
    at org.reflections8.Reflections.scan(Reflections.java:236)
    at org.reflections8.Reflections.<init>(Reflections.java:155)
    at org.reflections8.Reflections.<init>(Reflections.java:200)
    at org.reflections8.Reflections.<init>(Reflections.java:173)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at xxx.proj.Main.main(Main.java:25)
[main] WARN org.reflections8.Reflections - could not create Vfs.Dir from url. ignoring the exception and continuing
org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:118)
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:100)
    at org.reflections8.Reflections.scan(Reflections.java:278)
    at org.reflections8.Reflections.scan(Reflections.java:236)
    at org.reflections8.Reflections.<init>(Reflections.java:155)
    at org.reflections8.Reflections.<init>(Reflections.java:200)
    at org.reflections8.Reflections.<init>(Reflections.java:173)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at xxx.proj.Main.main(Main.java:25)

So How to solve 2nd problem?

cyw3 commented 3 years ago

OK, I try native-image-agent now.

But I encounter some problems:

  1. the same error as #2998. And after I change to use graalvm-jdk11, it works.
  2. org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]

Steps to reproduce the issue

java -agentlib:native-image-agent=config-output-dir=META-INF/native-image -jar debug.jar -m
ant clean jar
native-image -jar debug.jar -H:Name=debug -H:+ReportUnsupportedElementsAtRuntime -H:+ReportExceptionStackTraces -R:-RemoveNeverExecutedCode -H:-RemoveNeverExecutedCode -H:+AllowIncompleteClasspath
./debug -m
[main] WARN org.reflections8.Reflections - could not create Vfs.Dir from url. ignoring the exception and continuing
org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
  at org.reflections8.vfs.Vfs.fromURL(Vfs.java:118)
  at org.reflections8.vfs.Vfs.fromURL(Vfs.java:100)
  at org.reflections8.Reflections.scan(Reflections.java:278)
  at org.reflections8.Reflections.scan(Reflections.java:236)
  at org.reflections8.Reflections.<init>(Reflections.java:155)
  at org.reflections8.Reflections.<init>(Reflections.java:200)
  at org.reflections8.Reflections.<init>(Reflections.java:173)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  at xxx.proj.Main.main(Main.java:25)
[main] WARN org.reflections8.Reflections - could not create Vfs.Dir from url. ignoring the exception and continuing
org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
  at org.reflections8.vfs.Vfs.fromURL(Vfs.java:118)
  at org.reflections8.vfs.Vfs.fromURL(Vfs.java:100)
  at org.reflections8.Reflections.scan(Reflections.java:278)
  at org.reflections8.Reflections.scan(Reflections.java:236)
  at org.reflections8.Reflections.<init>(Reflections.java:155)
  at org.reflections8.Reflections.<init>(Reflections.java:200)
  at org.reflections8.Reflections.<init>(Reflections.java:173)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
  at xxx.proj.Main.main(Main.java:25)

So How to solve 2nd problem?

Corresponding to this code:

Reflections ref = new Reflections(scope);

it can not handle with scope. Why? How to solve it?

munishchouhan commented 3 years ago

@cyw3 So you are getting error with Java 8 and not with Java 11

cyw3 commented 3 years ago

OK, I try native-image-agent now. But I encounter some problems:

  1. the same error as #2998. And after I change to use graalvm-jdk11, it works.
  2. org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]

Steps to reproduce the issue

java -agentlib:native-image-agent=config-output-dir=META-INF/native-image -jar debug.jar -m
ant clean jar
native-image -jar debug.jar -H:Name=debug -H:+ReportUnsupportedElementsAtRuntime -H:+ReportExceptionStackTraces -R:-RemoveNeverExecutedCode -H:-RemoveNeverExecutedCode -H:+AllowIncompleteClasspath
./debug -m
[main] WARN org.reflections8.Reflections - could not create Vfs.Dir from url. ignoring the exception and continuing
org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:118)
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:100)
    at org.reflections8.Reflections.scan(Reflections.java:278)
    at org.reflections8.Reflections.scan(Reflections.java:236)
    at org.reflections8.Reflections.<init>(Reflections.java:155)
    at org.reflections8.Reflections.<init>(Reflections.java:200)
    at org.reflections8.Reflections.<init>(Reflections.java:173)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at xxx.proj.Main.main(Main.java:25)
[main] WARN org.reflections8.Reflections - could not create Vfs.Dir from url. ignoring the exception and continuing
org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:118)
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:100)
    at org.reflections8.Reflections.scan(Reflections.java:278)
    at org.reflections8.Reflections.scan(Reflections.java:236)
    at org.reflections8.Reflections.<init>(Reflections.java:155)
    at org.reflections8.Reflections.<init>(Reflections.java:200)
    at org.reflections8.Reflections.<init>(Reflections.java:173)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:490)
    at xxx.proj.Main.main(Main.java:25)

So How to solve 2nd problem?

Corresponding to this code:

Reflections ref = new Reflections(scope);

it can not handle with scope. Why? How to solve it?

@mcraj017

No, i also get this error with java 11.

munishchouhan commented 3 years ago

@cyw3 please provide the reproducer means class or jar to reproduce this

munishchouhan commented 3 years ago

@cyw3 just to confirm, if this application is working using java?

cyw3 commented 3 years ago

@cyw3 just to confirm, if this application is working using java?

Yes.

cyw3 commented 3 years ago

@mcraj017 Here is a demo with build.sh.

NativeImageTest.zip

#!/bin/bash
ant clean jar
java -agentlib:native-image-agent=config-output-dir=META-INF/native-image -jar release/test-debug.jar
ant clean jar
native-image -jar release/test-debug.jar -H:Name=test -H:+ReportUnsupportedElementsAtRuntime -H:+ReportExceptionStackTraces -R:-RemoveNeverExecutedCode -H:-RemoveNeverExecutedCode -H:+AllowIncompleteClasspath

./test
[main] WARN org.reflections8.Reflections - could not create Vfs.Dir from url. ignoring the exception and continuing
org.reflections8.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:118)
    at org.reflections8.vfs.Vfs.fromURL(Vfs.java:100)
    at org.reflections8.Reflections.scan(Reflections.java:278)
    at org.reflections8.Reflections.scan(Reflections.java:236)
    at org.reflections8.Reflections.<init>(Reflections.java:155)
    at org.reflections8.Reflections.<init>(Reflections.java:200)
    at org.reflections8.Reflections.<init>(Reflections.java:173)
    at test.Main.main(Main.java:10)
munishchouhan commented 3 years ago

@cyw3 Thanks for the reproducer

cyw3 commented 3 years ago

Hi, @christianwimmer . Any progress?

cyw3 commented 3 years ago

Hi, @mcraj017 , @christianwimmer Any progress?

andrejusc commented 2 years ago

@christianwimmer - was this resolved by any means or there is some known way to make it work for native image use case? Hitting similar issue today.

fniephaus commented 2 years ago

When I refactor your example Main into this:

public class Main {
    private static final Reflections ref = new Reflections("visitors");

    public static void main(String[] args) {
        for (Class<?> clz : ref.getTypesAnnotatedWith(VisitorAnnotation.class)) {
            System.out.println(clz);
        }
    }
}

I can do this:

$ ant clean jar
$ native-image -jar release/test-debug.jar -g -H:Name=test -H:+ReportUnsupportedElementsAtRuntime -H:+ReportExceptionStackTraces -R:-RemoveNeverExecutedCode -H:-RemoveNeverExecutedCode -H:+AllowIncompleteClasspath --initialize-at-build-time=test.Main,org.reflections8.Reflections,org.slf4j
$ ./test 
class visitors.TestVisitor

Also note that the binary is much smaller (18.38MB vs 45.33MB) now that org.reflections8.Reflections is initialized at build-time.

andrejusc commented 2 years ago

@fniephaus - While original issue here looks to be solvable by the way you provided above - I have additional iteration on top of that still to clarify related aspect to Reflections and Vfs and resource:/ interaction.

reflections-test.zip

Attached use case uses more recent org.reflections 0.10.2 and attempted to run using GraalVM 21.1 + JDK17

Initial steps:

$ mvn clean package
$ java -p target/libs:target/reflections-1.0-SNAPSHOT.jar --module com.example/com.example.Main

Such then produces output like such:

java -p target/libs:target/reflections-1.0-SNAPSHOT.jar --module com.example/com.example.Main
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
*** Getting reflections urls...
*** Conf urls: [jar:file:///dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/]
*** Vfs.dir: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/MANIFEST.MF
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/native-image/proxy-config.json
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/native-image/resource-config.json
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/native-image/jni-config.json
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/native-image/reflect-config.json
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/native-image/serialization-config.json
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/native-image/predefined-classes-config.json
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/native-image/native-image.properties
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/com/example/visitors/VisitorAnnotation.class
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/com/example/model/ForVisit.class
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/com/example/Main.class
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/maven/org.example/reflections/pom.xml
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/META-INF/maven/org.example/reflections/pom.properties
*** Vfs.file: /dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/module-info.class
*** Visiting class: class com.example.model.ForVisit

Now if I build that to native-image like such:

native-image -cp ./target/libs/reflections-0.10.2.jar:./target/libs/slf4j-api-1.7.32.jar:./target/libs/javassist-3.28.0-GA.jar -jar target/reflections-1.0-SNAPSHOT.jar

and try to invoke resulting reflections executable - I'm getting this:

$ ./reflections 
*** Getting reflections urls...
*** Conf urls: [resource:/]
org.reflections.ReflectionsException: could not create Vfs.Dir from url, no matching UrlType was found [resource:/]
either use fromURL(final URL url, final List<UrlType> urlTypes) or use the static setDefaultURLTypes(final List<UrlType> urlTypes) or addDefaultURLTypes(UrlType urlType) with your specialized UrlType.
    at org.reflections.vfs.Vfs.fromURL(Vfs.java:113)
    at org.reflections.vfs.Vfs.fromURL(Vfs.java:95)
    at com.example.Main.lambda$checkUrls$0(Main.java:24)
    at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1707)
    at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
    at com.example.Main.checkUrls(Main.java:21)
    at com.example.Main.main(Main.java:35)
*** Visiting class: class com.example.model.ForVisit

To my understanding resource: as protocol is supported by default within native-image. Do I need by some other means to make underlying reflections Vfs class/logic to be aware that such is handled, so it will not fail?

fniephaus commented 2 years ago

I refactored your example so that the Configuration is also static and with that, it works as expected:

package com.example;

import org.reflections.Reflections;
import org.reflections.scanners.Scanners;
import org.reflections.util.ConfigurationBuilder;
import org.reflections.vfs.Vfs;

import com.example.visitors.VisitorAnnotation;

public class Main {
    private static final Reflections ref = new Reflections("com.example.model",
        Scanners.FieldsAnnotated, Scanners.TypesAnnotated);
    private static final org.reflections.Configuration conf = ConfigurationBuilder.build("com.example.model",
                Scanners.FieldsAnnotated, Scanners.TypesAnnotated);

    private static void checkUrls() {
        System.out.println("*** Getting reflections urls...");
        var urls = conf.getUrls();
        System.out.println("*** Conf urls: " + urls);
        urls.stream().forEach(url -> {
            Vfs.Dir dir = null;
            try {
                dir = Vfs.fromURL(url);
                System.out.println("*** Vfs.dir: " + dir);
                for (Vfs.File file : dir.getFiles()) {
                    System.out.println("*** Vfs.file: " + file);
                }
            } catch (Exception e) {
                e.printStackTrace();
            }});
    }

    public static void main(String[] args) {
        checkUrls();
        for (Class<?> clz : ref.getTypesAnnotatedWith(VisitorAnnotation.class)) {
            System.out.println("*** Visiting class: " + clz);
        }
    }
}

Note that this will back the paths into your native executable, so this may reveal info about your build machine. I don't know what this reflections library is used for but it seems to be something that should be initialized at build-time. In this case, the binary size goes down from 284MB to 29MB when doing this. However, maybe there still is an issue with the virtual file system that provides these resources. Maybe @jovanstevanovic or @olpaw know more about that.

andrejusc commented 2 years ago

Proposed approach then changes such output line for native image case:

*** Conf urls: [jar:file:/dev/tmp/reflections/target/reflections-1.0-SNAPSHOT.jar!/]

which I'd still expect for native image to be in form of resource:/reference as it was before proposed change.

cyw3 commented 2 years ago

Thanks for your proposal:

  1. Need to set org.reflections8.Reflections, org.reflections8.vfs.Vfs and classes that use Reflections to --initialize-at-build-time;
  2. The instance of Reflections must be a field which is static final.

For the second point, I still need to modify the source code. Is there a non-intrusive solution that does not require modifying the source code?

fniephaus commented 2 years ago

For the second point, I still need to modify the source code. Is there a non-intrusive solution that does not require modifying the source code?

If you cannot change source code, I don't think there's much you can do to fix this properly.