Native Image: failing due to something with defineClass

[jeroenvandijk]

Trying Graalvm on the Uberjar

➜  closh git:(master) ../graalvm-ee-1.0.0-rc7/Contents/Home/bin/native-image -jar target/project.jar
Build on Server(pid: 36851, port: 56152)*
[project:36851]    classlist:   5,749.37 ms
[project:36851]        (cap):   1,642.43 ms
[project:36851]        setup:   2,631.58 ms
[project:36851]     analysis:  63,243.42 ms
error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Unsupported method java.lang.ClassLoader.defineClass(String, byte[], int, int) is reachable: The declaring class of this element has been substituted, but this element is not present in the substitution class
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Detailed message:
Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Unsupported method java.lang.ClassLoader.defineClass(String, byte[], int, int) is reachable: The declaring class of this element has been substituted, but this element is not present in the substitution class
To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time.
Trace:
    at parsing clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46)
Call path from entry point to clojure.lang.DynamicClassLoader.defineClass(String, byte[], Object):
    at clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:45)
    at clojure.core$get_proxy_class.invokeStatic(core_proxy.clj:288)
    at clojure.core$get_proxy_class.doInvoke(core_proxy.clj:276)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at closh.zero.frontend.rebel.main(Unknown Source)
    at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:163)
    at com.oracle.svm.core.code.CEntryPointCallStubs.com_002eoracle_002esvm_002ecore_002eJavaMainWrapper_002erun_0028int_002corg_002egraalvm_002enativeimage_002ec_002etype_002eCCharPointerPointer_0029(generated:0)

Error: Processing image build request failed

Trying again with the flag:

➜  closh git:(master) ../graalvm-ee-1.0.0-rc7/Contents/Home/bin/native-image -jar target/project.jar --report-unsupported-elements-at-runtime
Build on Server(pid: 36851, port: 56152)
[project:36851]    classlist:   2,635.53 ms
[project:36851]        (cap):     954.92 ms
[project:36851]        setup:   1,398.82 ms
[project:36851]   (typeflow):  37,104.00 ms
[project:36851]    (objects):   9,684.44 ms
[project:36851]   (features):     234.98 ms
[project:36851]     analysis:  47,955.07 ms
[project:36851]     universe:     942.61 ms
[project:36851]      (parse):   4,516.09 ms
[project:36851]     (inline):   5,945.88 ms
[project:36851]    (compile):  45,147.67 ms
[project:36851]      compile:  57,041.09 ms
[project:36851]        image:   5,677.81 ms
[project:36851]        write:   1,709.01 ms
[project:36851]      [total]: 117,442.72 ms

This seems to work? No errors, however:

➜  closh git:(master) ✗ ./project
Exception in thread "main" java.lang.NullPointerException
    at java.lang.Throwable.<init>(Throwable.java:250)
    at java.lang.Exception.<init>(Exception.java:54)
    at java.lang.RuntimeException.<init>(RuntimeException.java:51)
    at java.lang.NullPointerException.<init>(NullPointerException.java:60)
    at java.io.Reader.<init>(Reader.java:78)
    at org.jline.utils.InputStreamReader.<init>(InputStreamReader.java:135)
    at org.jline.utils.InfoCmp.loadDefaultInfoCmp(InfoCmp.java:609)
    at org.jline.utils.InfoCmp.lambda$static$3(InfoCmp.java:619)
    at org.jline.utils.InfoCmp$$Lambda$702/2004434588.get(Unknown Source)
    at org.jline.utils.InfoCmp.getLoadedInfoCmp(InfoCmp.java:558)
    at org.jline.terminal.impl.AbstractTerminal.parseInfoCmp(AbstractTerminal.java:174)
    at org.jline.terminal.impl.DumbTerminal.<init>(DumbTerminal.java:102)
    at org.jline.terminal.TerminalBuilder.doBuild(TerminalBuilder.java:350)
    at org.jline.terminal.TerminalBuilder.build(TerminalBuilder.java:219)
    at rebel_readline.jline_api$create_terminal.invokeStatic(jline_api.clj:62)
    at rebel_readline.jline_api$create_terminal.doInvoke(jline_api.clj:59)
    at clojure.lang.RestFn.invoke(RestFn.java:397)
    at closh.zero.frontend.rebel$_main.invokeStatic(rebel.clj:80)
    at closh.zero.frontend.rebel$_main.invoke(rebel.clj:79)
    at clojure.lang.AFn.applyToHelper(AFn.java:152)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at closh.zero.frontend.rebel.main(Unknown Source)
    at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:163)

[sundbp]

Tip: The graal folks are pretty responsive if you file issues (label them SubstrateVM for the right person's attention).

I've had a lot of help with getting various clojure things through native-image via issues.

On Tue, Oct 9, 2018 at 10:27 AM Jeroen van Dijk notifications@github.com wrote:

Trying Graalvm on the Uberjar

➜ closh git:(master) ../graalvm-ee-1.0.0-rc7/Contents/Home/bin/native-image -jar target/project.jar Build on Server(pid: 36851, port: 56152)* [project:36851] classlist: 5,749.37 ms [project:36851] (cap): 1,642.43 ms [project:36851] setup: 2,631.58 ms [project:36851] analysis: 63,243.42 ms error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Unsupported method java.lang.ClassLoader.defineClass(String, byte[], int, int) is reachable: The declaring class of this element has been substituted, but this element is not present in the substitution class To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time. Detailed message: Error: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Unsupported method java.lang.ClassLoader.defineClass(String, byte[], int, int) is reachable: The declaring class of this element has been substituted, but this element is not present in the substitution class To diagnose the issue, you can add the option --report-unsupported-elements-at-runtime. The unsupported element is then reported at run time when it is accessed the first time. Trace: at parsing clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:46) Call path from entry point to clojure.lang.DynamicClassLoader.defineClass(String, byte[], Object): at clojure.lang.DynamicClassLoader.defineClass(DynamicClassLoader.java:45) at clojure.core$get_proxy_class.invokeStatic(core_proxy.clj:288) at clojure.core$get_proxy_class.doInvoke(core_proxy.clj:276) at clojure.lang.RestFn.applyTo(RestFn.java:137) at closh.zero.frontend.rebel.main(Unknown Source) at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:163) at com.oracle.svm.core.code.CEntryPointCallStubs.com_002eoracle_002esvm_002ecore_002eJavaMainWrapper_002erun_0028int_002corg_002egraalvm_002enativeimage_002ec_002etype_002eCCharPointerPointer_0029(generated:0)

Error: Processing image build request failed

Trying again with the flag:

➜ closh git:(master) ../graalvm-ee-1.0.0-rc7/Contents/Home/bin/native-image -jar target/project.jar --report-unsupported-elements-at-runtime Build on Server(pid: 36851, port: 56152) [project:36851] classlist: 2,635.53 ms [project:36851] (cap): 954.92 ms [project:36851] setup: 1,398.82 ms [project:36851] (typeflow): 37,104.00 ms [project:36851] (objects): 9,684.44 ms [project:36851] (features): 234.98 ms [project:36851] analysis: 47,955.07 ms [project:36851] universe: 942.61 ms [project:36851] (parse): 4,516.09 ms [project:36851] (inline): 5,945.88 ms [project:36851] (compile): 45,147.67 ms [project:36851] compile: 57,041.09 ms [project:36851] image: 5,677.81 ms [project:36851] write: 1,709.01 ms [project:36851] [total]: 117,442.72 ms

This seems to work? No errors, however:

➜ closh git:(master) ✗ ./project Exception in thread "main" java.lang.NullPointerException at java.lang.Throwable.(Throwable.java:250) at java.lang.Exception.(Exception.java:54) at java.lang.RuntimeException.(RuntimeException.java:51) at java.lang.NullPointerException.(NullPointerException.java:60) at java.io.Reader.(Reader.java:78) at org.jline.utils.InputStreamReader.(InputStreamReader.java:135) at org.jline.utils.InfoCmp.loadDefaultInfoCmp(InfoCmp.java:609) at org.jline.utils.InfoCmp.lambda$static$3(InfoCmp.java:619) at org.jline.utils.InfoCmp$$Lambda$702/2004434588.get(Unknown Source) at org.jline.utils.InfoCmp.getLoadedInfoCmp(InfoCmp.java:558) at org.jline.terminal.impl.AbstractTerminal.parseInfoCmp(AbstractTerminal.java:174) at org.jline.terminal.impl.DumbTerminal.(DumbTerminal.java:102) at org.jline.terminal.TerminalBuilder.doBuild(TerminalBuilder.java:350) at org.jline.terminal.TerminalBuilder.build(TerminalBuilder.java:219) at rebel_readline.jline_api$create_terminal.invokeStatic(jline_api.clj:62) at rebel_readline.jline_api$create_terminal.doInvoke(jline_api.clj:59) at clojure.lang.RestFn.invoke(RestFn.java:397) at closh.zero.frontend.rebel$_main.invokeStatic(rebel.clj:80) at closh.zero.frontend.rebel$_main.invoke(rebel.clj:79) at clojure.lang.AFn.applyToHelper(AFn.java:152) at clojure.lang.AFn.applyTo(AFn.java:144) at closh.zero.frontend.rebel.main(Unknown Source) at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:163)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/dundalek/closh/issues/93, or mute the thread https://github.com/notifications/unsubscribe-auth/AADxJxJx8X5MaB3hpiB8ET-t-w_rZX1Kks5ujGv_gaJpZM4XSddT .

[dundalek]

This may already be fixed. Not sure whether it is already released or a building graal from source is still needed as the comment in the issue suggets.

[jeroenvandijk]

I've downloaded the latest enterprise version on the download site of Oracle "GraalVM based on JDK8, preview for macOS (1.0.0 RC7)" [1]

[1] https://www.oracle.com/technetwork/oracle-labs/program-languages/downloads/index.html

[dundalek]

It looks like 1.0.0 RC7 was released less than a week ago so the mentioned fix is probably in. Therefore there are likely other issues then.

So far I haven't heard any success stories from anyone in the community about running a bootstrapped clojure with graal. So it might be worth to hold off until some progress is made there.

[sundbp]

I've had several successes with clojure+graal. The tricky parts is if there are dependencies on native libraries and when there's lots of usage of Unsafe in dependencies (i.e. java libs, don't think I've seen it pop up in actual clojure code anytime).

It's not for the faint of heart, but clojure and graal itself isn't an issue - graal and situation X in general on the JVM is where there'll be constant progress as more and more people try their projects on it.

I may try to give it a spin later in the week if I get some free time.

On Tue, Oct 9, 2018 at 8:36 PM Jakub Dundalek notifications@github.com wrote:

It looks like 1.0.0 RC7 https://github.com/oracle/graal/releases/tag/vm-1.0.0-rc7 was released less than a week ago so the mentioned fix is probably in. Therefore there are likely other issues then.

So far I haven't heard any success stories from anyone in the community about running a bootstrapped clojure with graal. So it might be worth to hold off until some progress is made there.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/dundalek/closh/issues/93#issuecomment-428322729, or mute the thread https://github.com/notifications/unsubscribe-auth/AADxJ3ABZSLGAtOznYnl99WWVLCW4e3bks5ujPrCgaJpZM4XSddT .

[dundalek]

@sundbp I am also aware of people compiling native images of clojure applications. I think the issue is creating a native image of Clojure compiler itself due to the dynamic class loading. But the last information I could find on the topic is about 5 months old.

Thank you for your comments and any help is appreciated. It is such a cool tech, but sort of scary to dig into when things don't work out of the box :)

[sundbp]

Ah, right - I didn't take into consideration the compiler part itself. Now I understand the hesitation better :)

On Tue, Oct 9, 2018 at 9:27 PM Jakub Dundalek notifications@github.com wrote:

@sundbp https://github.com/sundbp I am also aware of people compiling native images of clojure applications. I think the issue is creating a native image of Clojure compiler itself due to the dynamic class loading. But the last information I could find on the topic is about 5 months old https://www.reddit.com/r/Clojure/comments/8fienj/native_clojure_with_graalvm/dy3zwjp/ .

Thank you for your comments and any help is appreciated. It is such a cool tech, but sort of scary to dig into when things don't work out of the box :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dundalek/closh/issues/93#issuecomment-428339287, or mute the thread https://github.com/notifications/unsubscribe-auth/AADxJ9bxb3r9phn6fQd91XbX-gq1vhGiks5ujQbKgaJpZM4XSddT .

[jeroenvandijk]

clojure 1.8 could be ok, but that's probably undesirable (according to this issue https://github.com/oracle/graal/issues/612)

[jeroenvandijk]

Below are two other references. [1] describes an example compiling clojure to graalVM (semi-succesfully). [2] describe limitations of GraalVM. It sounds the Dynamic class loading/unloading is the blocker. Not sure how this can be removed from Clojure and still have something working.

These limitations seem to indicate that a full blown version of Closh would be hard, but maybe a trimmed version with carefully selected dependencies could work. Maybe particularly useful for scripting, while keeping a slower, but more complete version, for longer shell sessions.

[1] https://www.innoq.com/en/blog/native-clojure-and-graalvm/ [2] https://github.com/oracle/graal/blob/master/substratevm/LIMITATIONS.md

[dundalek]

@jeroenvandijk These are great findings!

I think the only thing that closh depends on from 1.9 is spec. But it might be possible to use spec in 1.8 with spec backport.

[jeroenvandijk]

We've been a "long" time user of the backport. It has worked well for us. I can give it a try

[dundalek]

That would be great. I am very curious if it works or not.

[jeroenvandijk]

No luck:

clojure 1.8

clojure 1.10-0-RC1

By printing the Clojure version, I also found that GraalVM has filename bashed caching.. Had to rename the file. I also saw that the stacktrace didn't have line numbers the second time. So I wonder if individual runs are independent of each other.

[jeroenvandijk]

Ok maybe good news, I've been able to create a hello world executable

Output here

➜ closh git:(feature/bare-graal-client) ✗ boot -d boot/new new -t app -n myapp Retrieving maven-metadata.xml from https://repo.clojars.org/ Generating a project called myapp based on the 'app' template. ➜ myapp git:(feature/bare-graal-client) ✗ boot build Classpath conflict: org.clojure/clojure version 1.9.0 already loaded, NOT loading version 1.10.0-RC1 Compiling 1/1 myapp.core... Writing pom.xml and pom.properties... Adding uberjar entries... Writing myapp-0.1.0-SNAPSHOT-standalone.jar... Writing target dir(s)... ➜ myapp git:(feature/bare-graal-client) ✗ java -jar target/myapp-0.1.0-SNAPSHOT-standalone.jar Hello, World! ➜ myapp git:(feature/bare-graal-client) ✗ time java -jar target/myapp-0.1.0-SNAPSHOT-standalone.jar Hello, World! java -jar target/myapp-0.1.0-SNAPSHOT-standalone.jar 1.47s user 0.11s system 175% cpu 0.901 total ➜ myapp git:(feature/bare-graal-client) ✗ ../graalvm-ee-1.0.0-rc7/Contents/Home/bin/native-image -jar target/project.jar ➜ myapp git:(feature/bare-graal-client) ✗ ../../graalvm-ee-1.0.0-rc7/Contents/Home/bin/native-image -jar target/myapp-0.1.0-SNAPSHOT-standalone.jar Build on Server(pid: 97975, port: 61038)* [myapp-0.1.0-SNAPSHOT-standalone:97975] classlist: 5,917.43 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] (cap): 1,609.12 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] setup: 2,708.82 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] (typeflow): 6,156.22 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] (objects): 2,695.89 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] (features): 79.89 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] analysis: 9,107.05 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] universe: 365.64 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] (parse): 1,937.55 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] (inline): 2,729.59 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] (compile): 13,074.22 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] compile: 18,319.12 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] image: 1,784.07 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] write: 929.66 ms [myapp-0.1.0-SNAPSHOT-standalone:97975] [total]: 39,234.09 ms ➜ myapp git:(feature/bare-graal-client) ✗ ./myapp-0.1.0-SNAPSHOT-standalone Hello, World! ➜ myapp git:(feature/bare-graal-client) ✗ time ./myapp-0.1.0-SNAPSHOT-standalone Hello, World! ./myapp-0.1.0-SNAPSHOT-standalone 0.00s user 0.00s system 60% cpu 0.014 total

By itself nothing special, but what if we kept it very simple. No evalling just forwarding input to a backend and vice versa as described in #101. I'm guessing we would have no ClassLoader issues

[jeroenvandijk]

I've created something more than hello world in #105 :)

So far it seems that simple code works if you don't have any reflections. Compilation is pretty fast for a small project as well.

[jeroenvandijk]

In addition to the above, reflections warnings are not always found during compile time of the uberjar, and I'm not even sure i they are always found at runtime when running it as a jar. As an example, this type hint only cropped up in the binary, at runtime, with a cryptic error like no method with signature x.

[jeroenvandijk]

Probably useful to follow this project as it has overlapping challenges and interests with closh https://github.com/boot-clj/boot-native

[jeroenvandijk]

Should this one be closed? The sci approach in master is likely to make this redundant.

[dundalek]

I think so too.