I got JWM working with GraalVM to produce native binaries =)

[GavinRay97]

Tonsky, you held up your end of the deal -- now I get to hold up mine 😃

Video of fresh run, showing starting from nothing, compiling the JWM app, and then launching it (0:55 seconds in, compilation takes a while lol):

https://user-images.githubusercontent.com/26604994/134990377-b8da88f7-40bf-46ec-9007-2081294e09ae.mp4

Reproduction

Here's a zipped copy of the whole project, including the built files that contain the required GraalVM configuration:

• jwm-test.zip

This was done with GraalVM 21.3.0-dev, JDK 17:

λ java --version
openjdk 17 2021-09-14
OpenJDK Runtime Environment GraalVM CE 21.3.0-dev (build 17+35-jvmci-21.3-b02)
OpenJDK 64-Bit Server VM GraalVM CE 21.3.0-dev (build 17+35-jvmci-21.3-b02, mixed mode, sharing)

What I did was set up a basic Gradle Java app, using the GettingStarted.java example code:

• https://github.com/HumbleUI/JWM/blob/main/docs/GettingStarted.java

plugins {
    id 'application'
    id 'org.graalvm.buildtools.native' version '0.9.5'
}

repositories {
    mavenCentral()
    // GraalVM Native plugin, not on Maven yet
    // See: https://graalvm.github.io/native-build-tools/0.9.5/gradle-plugin.html
    gradlePluginPortal()
    // JWM local jar file, at "app/lib/jwm-main.jar"
    flatDir {
        dirs 'lib'
    }
}

dependencies {
    implementation name: 'jwm-main'
}

application {
    mainClass = 'jwm.test.Application'
}

graalvmNative {
    binaries {
      main {
          verbose = true
      }
    }
}

Trying to build the JWM app and run it normally will throw a segfault.

To fix this, you have to first run the "agent" to profile the GraalVM app and determine what configuration it needs for things like JNI, dynamic behavior, etc:

• https://graalvm.github.io/native-build-tools/0.9.5/gradle-plugin.html#_reflection_support_and_running_with_the_native_agent

./gradlew -Pagent run # Runs on JVM with native-image-agent.
./gradlew -Pagent nativeCompile # Builds image using configuration acquired by agent.

This is broken though, at least for me. It was generating the configuration in: app\build\native\agent-output

But I could see when it was compiling that it was only sourcing config from: -H:ConfigurationFileDirectories=app\build\native\generated\generateResourcesConfigFile

So I just copied all the files from app\build\native\agent-output to app\build\native\generated\generateResourcesConfigFile and then it worked:

If it's useful at all, below is an output of the compilation task with verbose logging:

Click to view

C:\Users\rayga\Projects\tmp\jwm-test λ gradlew.bat -Pagent nativeCompile > Task :app:nativeCompile [native-image-plugin] Args are: [-cp, C:\Users\rayga\Projects\tmp\jwm-test\app\build\libs\nativecompile-classpath.jar, --no-fallback, --verbose, -H:Path=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile, -H:Name=app, -H:ConfigurationFileDirectories=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\generated\generateResourcesConfigFile, --allow-incomplete-classpath, -H:Class=jwm.test.Application] Executing [ 'C:\GraalVM\graalvm-ce-java17-21.3.0-dev\bin\java.exe' \ -XX:+UseJVMCINativeLibrary \ -XX:+UseParallelGC \ -XX:+UnlockExperimentalVMOptions \ -XX:+EnableJVMCI \ -Dtruffle.TrustAllTruffleRuntimeProviders=true \ -Dtruffle.TruffleRuntime=com.oracle.truffle.api.impl.DefaultTruffleRuntime \ -Dgraalvm.ForcePolyglotInvalid=true \ -Dgraalvm.locatorDisabled=true \ -Dsubstratevm.IgnoreGraalVersionCheck=true \ --add-exports=java.base/com.sun.crypto.provider=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.access.foreign=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.event=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.loader=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.logger=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.module=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.org.objectweb.asm=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.org.xml.sax.helpers=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.perf=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.ref=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.util.xml.impl=ALL-UNNAMED \ --add-exports=java.base/jdk.internal.util.xml=ALL-UNNAMED \ --add-exports=java.base/sun.invoke.util=ALL-UNNAMED \ --add-exports=java.base/sun.nio.ch=ALL-UNNAMED \ --add-exports=java.base/sun.reflect.annotation=ALL-UNNAMED \ --add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED \ --add-exports=java.base/sun.reflect.generics.repository=ALL-UNNAMED \ --add-exports=java.base/sun.reflect.generics.tree=ALL-UNNAMED \ --add-exports=java.base/sun.security.jca=ALL-UNNAMED \ --add-exports=java.base/sun.security.provider=ALL-UNNAMED \ --add-exports=java.base/sun.security.util=ALL-UNNAMED \ --add-exports=java.base/sun.text.spi=ALL-UNNAMED \ --add-exports=java.base/sun.util.calendar=ALL-UNNAMED \ --add-exports=java.base/sun.util.locale.provider=ALL-UNNAMED \ --add-exports=java.base/sun.util.resources=ALL-UNNAMED \ --add-exports=java.xml.crypto/org.jcp.xml.dsig.internal.dom=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.aarch64=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.amd64=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.site=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.code.stack=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.code=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.common=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.aarch64=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot.amd64=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.hotspot=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.meta=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.runtime=ALL-UNNAMED \ --add-exports=jdk.internal.vm.ci/jdk.vm.ci.services=ALL-UNNAMED \ --add-exports=jdk.jfr/jdk.jfr.events=ALL-UNNAMED \ --add-exports=jdk.jfr/jdk.jfr.internal.consumer=ALL-UNNAMED \ --add-exports=jdk.jfr/jdk.jfr.internal.handlers=ALL-UNNAMED \ --add-exports=jdk.jfr/jdk.jfr.internal.jfc=ALL-UNNAMED \ --add-exports=jdk.jfr/jdk.jfr.internal=ALL-UNNAMED \ -Xss10m \ -Xms1g \ -Xmx14g \ -Duser.country=US \ -Duser.language=en \ -Djava.awt.headless=true \ -Dorg.graalvm.version=21.3.0-dev \ -Dorg.graalvm.config=CE \ -Dcom.oracle.graalvm.isaot=true \ -Djava.system.class.loader=com.oracle.svm.hosted.NativeImageSystemClassLoader \ -Xshare:off \ -Djdk.internal.lambda.disableEagerInitialization=true \ -Djdk.internal.lambda.eagerlyInitialize=false \ -Djava.lang.invoke.InnerClassLambdaMetafactory.initializeLambdas=false \ '-javaagent:C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\svm.jar' \ -cp \ 'C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\objectfile.jar;C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\pointsto.jar;C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\builder\svm.jar' \ --module-path \ 'C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\truffle\truffle-api.jar' \ 'com.oracle.svm.hosted.NativeImageGeneratorRunner$JDK9Plus' \ -imagecp \ 'C:\Users\rayga\Projects\tmp\jwm-test\app\build\libs\nativecompile-classpath.jar;C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\library-support.jar' \ '-H:Path=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile' \ -H:FallbackThreshold=0 \ '-H:Path=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile' \ -H:Name=app \ '-H:ConfigurationFileDirectories=C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\generated\generateResourcesConfigFile' \ -H:+AllowIncompleteClasspath \ -H:Class=jwm.test.Application \ '-H:CLibraryPath=C:\GraalVM\graalvm-ce-java17-21.3.0-dev\lib\svm\clibraries\windows-amd64' ] [app:28380] classlist: 2,552.43 ms, 0.96 GB [app:28380] (cap): 5,273.75 ms, 0.96 GB [app:28380] setup: 7,129.85 ms, 0.96 GB [app:28380] (clinit): 221.34 ms, 2.37 GB [app:28380] (typeflow): 3,175.61 ms, 2.37 GB [app:28380] (objects): 5,393.20 ms, 2.37 GB [app:28380] (features): 652.02 ms, 2.37 GB [app:28380] analysis: 10,019.95 ms, 2.37 GB [app:28380] universe: 973.36 ms, 2.37 GB [app:28380] (parse): 560.62 ms, 2.37 GB [app:28380] (inline): 815.50 ms, 2.37 GB [app:28380] (compile): 7,547.62 ms, 4.67 GB [app:28380] compile: 9,666.75 ms, 4.67 GB [app:28380] image: 1,291.57 ms, 4.67 GB [app:28380] write: 2,160.76 ms, 4.67 GB [app:28380] [total]: 34,818.08 ms, 4.67 GB # Printing build artifacts to: C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile\app.build_artifacts.txt [native-image-plugin] Native Image written to: C:\Users\rayga\Projects\tmp\jwm-test\app\build\native\nativeCompile

[GavinRay97]

I was also able to export a C ABI function from Java, and compile my JWM app as a .dll then call it from C++!! 😲 😍

graalvmNative {
    binaries {
        main {
            sharedLibrary = true
        }
    }
}

See gist for Application.java code:

• https://gist.github.com/GavinRay97/3b62127e1683b846d42a490fc73bc908

  @CEntryPoint(name = "displayEntrypoint")
  @CEntryPointOptions(prologue = CEntryPointSetup.EnterCreateIsolatePrologue.class, epilogue = CEntryPointSetup.LeaveTearDownIsolateEpilogue.class)
  static void display(long hwnd) {
      App.init();
  
      WindowWin32 window = new WindowWin32();
      window.setTitle("JWM Window");
      window.setWindowSize(250, 250);
      window.setEventListener(new EventHandler(window));
      window.setVisible(true);
      window.requestFrame();
      // Try reparent
      window.winSetParent(hwnd);
  
      App.start();
  }
  
  interface DisplayFunctionPointer extends CFunctionPointer {
      @InvokeCFunctionPointer
      void display(long hwnd);
  }
  
  private static final CEntryPointLiteral<DisplayFunctionPointer> displayCallback =
      CEntryPointLiteral.create(Application.class, "display");

Compiling this exports a header and a .dll:

[SHARED_LIB]
app.dll

[HEADER]
graal_isolate.h
app.h
graal_isolate_dynamic.h
app_dynamic.h

[IMPORT_LIB]
app.lib

The generated app.h contains the C definition for the exported @CEntryPoint() function:

// app.h
#ifndef __APP_H
#define __APP_H

#include <graal_isolate.h>

#if defined(__cplusplus)
extern "C" {
#endif

void displayEntrypoint(long long int);

int run_main(int argc, char** argv);

void vmLocatorSymbol(graal_isolatethread_t* thread);

#if defined(__cplusplus)
}
#endif
#endif

And if we look at the app.dll in Dependencies.exe, we can see the export here:

Writing a small test which uses a hardcoded window HWND:

// test.cpp
#include "app.h"
#pragma comment(lib, "app") // hacky -Lapp because I'm lazy

int main()
{
    long testHWND = 0x3530ae8;
    displayEntrypoint(testHWND);
    return 1;
}

Then running it -- the Graal-compiled .dll containing our JWM app is successfully started, and the exported function that attaches to another window is able to be called from our C++ program!!

We have liftoff! With all of this in place -- it confirms it is possible to write VST plugins in JVM languages using JWM + Graal! 💯 🔥 🙌

https://user-images.githubusercontent.com/26604994/135004266-5744dae5-32e6-423c-b542-c04baaafb26e.mp4

[i10416]

@GavinRay97

Great! I succeeds to run native image on both Windows and Linux thanks to information in this issue! https://github.com/HumbleUI/JWM/pull/160

[Hilal-Anwar]

That is so cool.

[GavinRay97]

@i10416 Awesome!! I saw you made a great PR! 🙌

Maybe soon the GraalVM checkboxes here will be green? 😅

[tonsky]

Looks great! I wasn’t able to run it on macOS, but Windows worked!

[GavinRay97]

@tonsky 🤔 I assume you probably checked this -- but there I think should be only one difference in the config files between OS'es and that would be the resource-config.json (plus manually copying the shared-lib into the same directory as the built binary)

// app\build\native\agent-output\run\resource-config.json
{
  "resources":{
  "includes":[
    {"pattern":"\\Qjwm.version\\E"}, 
    {"pattern":"\\Qjwm_x64.dll\\E"}
  ]},
  "bundles":[]
}

I suppose this should include the .dylib for Mac and the .so for Linux

This gets generated by running the "profiling agent" (native-image-agent) that instruments and records what the app accesses during it's lifetime:

• https://docs.oracle.com/en/graalvm/enterprise/19/guide/reference/native-image/tracing-agent.html

[tonsky]

Yeah, I did that. I think the problem lies in the fact that macOS requires all windows to be manipulated from the main thread of the app, so I have to write a C program that call JVM program etc. Probably doable, but a lot of stuff to figure out, and I am currently focused elsewhere.

I hope pick it up one day for sure, for now, I am glad you are unblocked. Let me know if anything else is missing in JWM

[GavinRay97]

Ah. Was this with the attempted re-parenting or just a basically empty app? Curious if it runs on Mac at all. And thanks a ton -- really appreciate it! 🙏

There is a LOT of neat stuff that can be built out with this now, for native apps where you would have before used C++ or Rust etc.

Maybe it's even useful for this?

• https://tonsky.me/blog/clojure-ui/

👀

[tonsky]

Without reparenting. Yes I plan to leverage this eventually for Clojure UI