This issue is to discuss possible support for Graal Native Image, and how that might be achieved.

Of late, I have been using Graal Native Image more and more, often along with dbus-java, particularly on Linux. For example, if you add PicoCLI it makes Java a great language for writing DBus based command line utilities that are easy to distribute, fast to start up and use a lot less memory. You can even totally statically link (e.g. with libc or musl). Who'd have thought it.

The main challenge to using native image, is providing the reflection (and other) meta-data to help Graal in it's tasks of examining every possible code path. This has always been a bit painful, but has improved of late with things such as the meta-data repository and better tools to automate this. Lots of libraries now either include Graal meta-data, or there are rebuilds of libraries that have had meta-data added (Quarkus etc).

Getting good Graal support in dbus-java itself is going to involved a few different tasks. Its mainly about some resource files, and a new tool.

Checking Dependencies

To have good support, all 3rd party dependencies will need to be checked if they require meta-data, and if they provide it. Either themselves or via an external means. I know JNA has it, other dependencies (in transports I think primarily) will need to be checked. The pure Java (17) one I don't think requires any additional meta-data.

Library Meta-data

Then there is the library itself. Adding meta-data to the project is done by adding .json files to src/main/resources/META-INF/native-image/[xxxxxx]/yyyyy.json, where xxxxxx is a project name (e.g. dbus-java) and yyyyyy is the one of the classes of meta-data.

You can use the native image tracing agent, to generate the meta-data at run-time, but this can generate a lot of platform specific cruft that often is not required. It is very helpful though, and we've found fine in production if a brute-force approach is acceptable.

The classes of meta-data are ..

jni-config.json - contains data about JNI methods called and libraries used
predefine-classes-config.json - not sure about this one, i've not yet seen it populated
proxy-config.json - data about Proxy implementations (dbus-java uses this one)
resource-config.json - data about classpath resources loaded. There are sometimes surprising things loaded as resources.
relect-config.json - data about things that need to be reflectable (dbus-java uses this one)
serialization-config.json - data about Serializable usages

So far, I've only discovered two types of meta-data that are needed for the basic library. proxy-config.json and reflect-config.json. That is not to say there are not others, but this is all i've needed so far.

proxy-config.json

[
    {
        "interfaces": [
            "org.freedesktop.dbus.interfaces.DBus"
        ]
    },
    {
        "interfaces": [
            "org.freedesktop.dbus.interfaces.Properties"
        ]
    },
    {
        "interfaces": [
            "org.freedesktop.dbus.interfaces.Introspectable"
        ]
    }
]

reflect-config.json

[
{
  "name":"org.freedesktop.dbus.interfaces.DBus$NameAcquired",
  "queryAllDeclaredConstructors":true,
  "methods":[{"name":"<init>","parameterTypes":["java.lang.String","java.lang.String"] }]
},
{
  "name":"org.freedesktop.dbus.interfaces.Introspectable",
  "allDeclaredClasses":true,
  "queryAllDeclaredMethods":true
},
{
  "name":"org.freedesktop.dbus.interfaces.Peer",
  "allDeclaredClasses":true,
  "queryAllDeclaredMethods":true
}
]

Annotation Processor or Code Generator

Lastly, it would be great if dbus-java also provided the tools for users to generate meta-data for their own DBusInterface implementations, and other DBus structures.

For example, DBusInterface implementations must be both declared in proxy-config.json and must be fully reflectable for dbus-java to function. Struct must also be fully reflectable.

This could be done in InterfaceCodeGenerator, with it generating Graal meta-data along with annotated Java source, but I feel this would be better solved using a separate Annotation Processor, applied at a later time. This means hand-written dbus-java interfaces and structures would also be handled.

This is how PicoCLI solves Graal integration, by looking for it's own annotations and generating the appropriate json resources at build time. Annotation processors are a Java feature, and tools like Maven and others have well documented features for enabling processors.

However, an "Annotation Processor" is exactly that. It finds annotations. In dbus-java, annotations for example on a DBusInterface are optional. For this reason, it may be better to have a new annotation, e.g. @DBusNative that tags either a DBusInterface or Struct as a candidate for generation of Graal meta-data.

Risks

None really. Any changes should be entirely non-invasive, and not require any particular version of Java. There would only be a single new annotation.

Some transport providers may require either additional meta-data.

Graal native image developers should be by now well used to tweaking or providing additional meta-data while everybody catches up.

Conclusion

Graal native image really is awesome. It is going to be a significant part of Javas future, so it would be great to get 1st class support here.

If the idea sounds good to you, I will at some point get a PR together. I have most of the parts to achieve all of the above, it just needs bringing together in dbus-java itself.

I don't have real opinion about Graal. I know it exists but I never really cared about it. For me it is some sort of black magic to solve a problem about Java in general which you would not have to solve if you would use a native (non-interpreted) language like Rust.

Anyway, I'm also not against adding support for this as long as it does not create additional effort for every new dbus-java version (while releasing or for keeping the bunch of json files up-to-date and compatible with future Graal versions).

The best solution would be if dbus-java uses the same approach to be Graal compatible as the developer who uses dbus-java itself. I want to have the change as minimal invasive as possible. Adding some annotations is fine.

What I want to avoid is to have some special logic for e.g. populating proxy-config.json. All those Graal configs should be auto-generated on build using annotations or something like that. This will ensure that the json files will be updated in the future when e.g. when usage of Proxy changed.

One may consider to support multiple annotations. What I'm thinking of is something like: Either you annotate your class with a new annotation (e.g. @DBusGraal, I would avoid using @DBusNative as it sounds like something done with DBus itself natively) or your class has to be annotated with the "old" @DBusInterfaceName annotation. Both should be considered for building the json. If both are present it is also fine but should not change the outcome.

I don't know which options we have, maybe we can also eliminate the need for @DBusGraal if there is some way to detect if a class or interface implement/extends DBusInterface (all of those would be candidates for reflect-config.json).

If you have some 'experimental' branch to play with, you may share it.

I don't have real opinion about Graal. I know it exists but I never really cared about it. For me it is some sort of black magic to solve a problem about Java in general which you would not have to solve if you would use a native (non-interpreted) language like Rust.

I won't try to sell you Graal :) Well, Graal Native Image anyway. "Graal" covers a lot more than AOT native compilation.

The best solution would be if dbus-java uses the same approach to be Graal compatible as the developer who uses dbus-java itself. I want to have the change as minimal invasive as possible. Adding some annotations is fine.

Indeed. Once written, there would be nothing stopping dbus-java itself using its own annotation processor to generate it's own meta-data.

One may consider to support multiple annotations. What I'm thinking of is something like: Either you annotate your class with a new annotation (e.g. @DBusGraal, I would avoid using @DBusNative as it sounds like something done with DBus itself natively) or your class has to be annotated with the "old" @DBusInterfaceName annotation. Both should be considered for building the json. If both are present it is also fine but should not change the outcome.

That sound fine. I am fairly sure that annotation processors cannot find un-annotated interfaces. I'll look into though. If it can't, you'd have to parse Java source itself. Of course there are plenty of libraries out there to do so, but it just seems overkill.

If you have some 'experimental' branch to play with, you may share it.

Nothing ready yet, but it shouldn't take long. Watch this space.

Diving into this a bit more, the requirement to have the annotation processor be used to generate the default Graal meta-data is what is the hard part of this task.

Ideally, the following need to be true.

Any new annotation should be in dbus-java-core
The annotation processor shouldn't be in the core, to keep it pure and avoid additional dependencies the processor requires (a JSON library that could be excluded by just generating strings, and the Java annotation processor API).
The annotation processor needs access to the DBusGraal annotation class.
The graal meta-data resources should be in dbus-java-core.
The graal meta-data resources should be generated the same way as user-code, i.e. with the annotation processor.
This implies the annotation processor needs to be compiled before dbus-java-core. It can't though, because to compile the annotation processor dbus-java-core must be compiled.
All of this must work within JPMS constraints

Right now, I can think of a few ways to solve this, none of them are great.

Have a "root" library module that contains just the annotation. The processor module and dbus-java-core can then depend on this. This is probably the cleanest and easiest solution. However, I suspect you will hate this. I do. There will be a new always-present dependency, with a single class in it.
Keep the annotation in the annotation processor library, and have dbus-java-core depend on this. The dependency can be declared as <optional> at the Maven level and static in module-info.java. The annotation only requires SOURCE retention, so this should mean there are no additional run-time requirements. A user simply includes dbus-java-core as normal, and won't get the annotation unless they ask for it.
Keep the annotation and the annotation processor in dbus-java-core. Make the annotation processor API maven dependency and module-info entry optional so they do not become runtime requirements. The compilation of dbus-java-core must be done in two phases. First time without annotation processing, the 2nd time with. Note, you have to do this anytime you wish to both build and use an annotation processor using Maven anyway.
Don't use the annotation processor at all. Just provide the fixed meta-data as posted above in dbus-java-core (well, an enhanced version of it). The annotation processor can then just be a separate module which depends on the core. FWIW, the format of these JSON files hasn't really changed for several years and seems unlikely to. The content (i.e. the core DBUS APIs) seem unlikely to change much either. I understand though, it's possible additional maintenance you don't really want.

I'll keep thinking about this. The annotation processor itself is pretty much written. I just need to get this arrangement right.

Interesting investigation and also great timing as I just begun to take a look at this.

I have to admit that I'm also not happy with either solution you mentioned. Doing all that boilerplate stuff for this will be quite painful when releasing a new version. Using a "static" configuration for Graal can be bad as well (missing changes when refactoring, missing new interfaces, etc).

Also providing a new annotation will always require existing code to be changed to get compatible with the Graal stuff. I'm not a friend of adding code just to please some toolkit.

Anyway. As already said, I took some deeper look into this and into the idea of "finding every interface which extends DBusInterface". My first intention was to create a javac plugin, but that seems to be a dead end. The documentation is bad and starting with JDK 9 one does not even get access to the required Plugin class (because of JPMS). Starting with more recent Java release the access to "sun" packages got even worse. So this is not a real good solution as far as I can tell.

I then googled around a bit and found a library called "spoon". Spoon can parse Java source and create AST which can be analyzed. It also allows manipulating the code, but that is nothing I want to do. I played around a bit and this is my first POC:

<dependency>
    <groupId>fr.inria.gforge.spoon</groupId>
    <artifactId>spoon-core</artifactId>
    <version>11.0.0</version>
</dependency>

package com.github.hypfvieh.dbus.graal;

import spoon.Launcher;
import spoon.SpoonAPI;
import spoon.reflect.declaration.CtType;
import spoon.reflect.visitor.filter.AbstractFilter;

import java.util.LinkedHashSet;
import java.util.Set;

public final class CreateGraalConfig {

    private static final Set<String> CANDIDATES = Set.of(
        "org.freedesktop.dbus.interfaces.DBusInterface",
        "org.freedesktop.dbus.Container",
        "org.freedesktop.dbus.interfaces.DBusSigHandler");

    private CreateGraalConfig() {}

    public static void main(String[] _args) {

        SpoonAPI spoon = new Launcher();
        spoon.addInputResource("../dbus-java-examples/src/main/java/");
        spoon.getEnvironment().setComplianceLevel(17);
        spoon.buildModel();

        Set<String> found = new LinkedHashSet<>();

        spoon.getModel().getElements(new AbstractFilter<CtType<?>>() {
            @Override
            public boolean matches(CtType<?> _element) {
                if (_element.getSuperInterfaces().stream()
                    .anyMatch(e -> CANDIDATES.contains(e.getQualifiedName()))) {
                    found.add(_element.getQualifiedName());
                    return true;
                }
                return false;
            };
        });

        System.out.println(found);
    }
}

This code utilizes spoon to find all classes and interfaces extending one of the interfaces found in "CANDIDATES". All found interfaces will be printed to STDOUT at the end.

My idea is to use this to find all we need to create proper JSON files. To do that, I would prefer to add another module to the project containing the code and dependencies required for this step. This means everyone who is not interested in this Graal stuff will not even notice about that change. Also every additional dependency (e.g. spoon or any JSON library like Jackson) will be part of another module/library and will not pollute every project using dbus-java-core.

The only problem I see right now is the dependency between the new module (dbus-java-graal) and dbus-java-core. If there is a dependency (e.g. to include all required interface names as classes instead of strings to be refactoring safe) we have the same circular issue like you stated above. Dbus-java-graal would require dbus-java-core and vice versa. Using the interface names as string would solve this, but will again introduce problems when the interface names change in future (even though there are no plans to do this).

What do you think?

Oh ok, nice. It would certainly be nice if no annotation was needed at all.

But yes, as you say, as soon as you use real Class<?> references, it gets harder. Do you really need to use those real class references though?

The number actually used by dbus-java itself is quite small. If the CreateGraalConfig tool were invoked by maven as part of the normal build process, and if it were to make the build fail completely if those string class names are wrong, then at the absolute worse this would be highlighted build time. Not perfect, but at least it would mean everything could be totally separate.

One option would be to stick to those String constants, so no circular dependency would be needed.

Another idea I just got, is to use maven-enforcer-plugin:

Add a text file containing the class names required for the CreateGraalConfig tool
- the tool reads this file to get the required information currently hardcoded
Using maven-enforce-plugin with beanshell-rule to read the content of that text file and ensure that there are corresponding classes found in dbus-java-core.
- This means: read the file, create file pathes of every line, add the .java suffix and check if the file in dbus-java-core/src/main/java exists. If not: fail

It may be hacky somehow, but will ensure that no refactoring will break anything because the Maven build will fail. I updated my sample code and the pom. You can find everything in the graal branch.

Maybe you can use this as starting point. I know that you can also query the spoon-results to get method names, constructors and parameters for those. So I guess all information required for the Graal config files are already available. The last missing piece would be to execute CreateGraalConfig before/after compiling dbus-java-core (e.g. exec-maven-plugin) so the required meta data will be present when the JAR is build.

Yup, that looks like it will do that job. I'll take your branch and add my Json generating code and see what happens.

Just a little progress report ....

CreateGraalConfig is complete and generating the JSON, with command line options and help. Uses Picocli (i figured one more dependency wouldn't hurt here, and picocli is awesome).
There is a Maven plugin that requires zero options or configuration (although supports same config as CLI tool)
The code is reusable for others to potentially build into other tools (Ant, Gradle?). Uses builder pattern.
A detailed documentation page is written.

As a bonus, I added a Maven profile to dbus-java-graal-native that will generate a create-graal-config.exe. Once Graal is installed (and you have a C compiler installed), you can ..

mvn package -P native-image

to create this standalone executable. Spoon has quite a few dependencies, and this is a nice way to bundle them all up. Then a user just needs to create-graal-config [options ..] /path/to/input.

GitHub actions allows for Graal compilation on open source projects, so this tool could be automatically built and released. (although I doubt it will change much). I have zero idea how to do this though, I just know it's possible.

There is still a little to do.

Testing! I have several projects that are currently manually defining dbus-java metadata that can be used to prove it.
Unit tests for the JSON generation.
Standard Maven "Mojo" help. I need to figure out how this integrates with the rest of the dbus-java documenation / site.

Sounds good so far.

I don't think I will add a automatic Graal build. When someone needs that feature, they should use the maven plugin or run the util using java or maven. I don't like creating binaries for random platforms. I had that in the past with native code in dbus-java and don't want to do anything like that again. There will always be someone who wants to have first class support for their weird used architecture or OS (e.g. using arbitrary ARM/RISC based systems, random nix OSes starting with Linux+glibc over Linux+uclibc/dietlibc, BSD... until you get the various MacOS flavors and don't forget about the Windows folks). Simply a nightmare.

I'll await your PR as soon as you are sure that everything "fits".

hypfvieh / dbus-java

Graal Native Image Support #263