search

nicokosi / hubstats

🧪 Command-line for GitHub pull request statistics
5 stars 0 forks source link

Generate executable via GraalVM #12

Closed nicokosi closed 5 years ago

nicokosi commented 6 years ago

This interesting blog post made me want to try using GraalVM in order to generate hubstats' executable.

nicokosi commented 6 years ago

Attempt #1

From fat JAR, with Dockerized GraaVM 1.0.0-rc1: visibly hit similar issue than https://github.com/oracle/graal/issues/385 / https://github.com/oracle/graal/issues/375, related to "static initializers" limitation? 😭 : 

Step 4/15 : RUN native-image   -jar hubstats-0.1.0-SNAPSHOT-standalone.jar   -H:+ReportUnsupportedElementsAtRuntime   hubstats.core
 ---> Running in e7f911774bd4
Build on Server(pid: 11, port: 26681)*
   classlist:   3,159.26 ms
       (cap):   1,485.02 ms
       setup:   2,563.80 ms
    analysis:  10,109.06 ms
fatal error: java.lang.NullPointerException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
    at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
    at com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:398)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:240)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:337)
    at com.oracle.svm.hosted.server.NativeImageBuildServer.executeCompilation(NativeImageBuildServer.java:378)
    at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$processCommand$8(NativeImageBuildServer.java:315)
    at com.oracle.svm.hosted.server.NativeImageBuildServer.withJVMContext(NativeImageBuildServer.java:396)
    at com.oracle.svm.hosted.server.NativeImageBuildServer.processCommand(NativeImageBuildServer.java:312)
    at com.oracle.svm.hosted.server.NativeImageBuildServer.processRequest(NativeImageBuildServer.java:256)
    at com.oracle.svm.hosted.server.NativeImageBuildServer.lambda$serve$7(NativeImageBuildServer.java:216)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
    at com.oracle.graal.pointsto.ObjectScanner.scanField(ObjectScanner.java:113)
    at com.oracle.graal.pointsto.ObjectScanner.doScan(ObjectScanner.java:263)
    at com.oracle.graal.pointsto.ObjectScanner.finish(ObjectScanner.java:307)
    at com.oracle.graal.pointsto.ObjectScanner.scanBootImageHeapRoots(ObjectScanner.java:78)
    at com.oracle.graal.pointsto.ObjectScanner.scanBootImageHeapRoots(ObjectScanner.java:60)
    at com.oracle.graal.pointsto.BigBang.checkObjectGraph(BigBang.java:581)
    at com.oracle.graal.pointsto.BigBang.finish(BigBang.java:552)
    at com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:653)
    at com.oracle.svm.hosted.NativeImageGenerator.lambda$run$0(NativeImageGenerator.java:381)
    at java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1386)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Error: Processing image build request failed
nicokosi commented 6 years ago

Attempt #2

With locally built substratevm from commit https://github.com/oracle/graal/commit/c101571659966f5e5069173c2c63f2103da2aaa0, native-image hangs: 

$> JAVA_HOME=~/Downloads/labsjdk1.8.0_161-jvmci-0.42/Contents/Home ../graal/substratevm/native-image -jar target/hubstats-0.1.0-SNAPSHOT-standalone.jar hubstats.core
Build on Server(pid: 18933, port: 55103)*
   classlist:   2,744.32 ms
       (cap):   1,531.16 ms
       setup:   2,401.40 ms

Similar issue: https://github.com/oracle/graal/issues/411.

nicokosi commented 6 years ago

Give a try using arguments like:

native-image -H:+ReportUnsupportedElementsAtRuntime -J-Xmx3G -J-Xms3G -jar ./target/hubstats-0.1.0-SNAPSHOT-standalone.jar hubstats.core

Cf. https://www.astrecipes.net/blog/2018/07/20/cmd-line-apps-with-clojure-and-graalvm.

nicokosi commented 6 years ago

See the related GraalVM issue I have opened: https://github.com/oracle/graal/issues/420.

nicokosi commented 6 years ago

Related resources:

nicokosi commented 5 years ago

This comment on Clojure reflection / GraalVM limitation can also be useful.

nicokosi commented 5 years ago

Two big news:

nicokosi commented 5 years ago

Oops, hubstats's executable still cannot be generated with GraalVM 1.0.0 RC 11, visibly. :boom: 

Compiling hubstats.options
Created /home/nkosinski/perso/hubstats/target/hubstats-0.1.0-SNAPSHOT.jar
Created /home/nkosinski/perso/hubstats/target/hubstats-0.1.0-SNAPSHOT-standalone.jar
Build nicokosi/hubstats's executable from JAR via GraalVM:
1.0.0-rc11: Pulling from oracle/graalvm-ce
Digest: sha256:ab6b50e27cab6bca72e1d35c0b31f9f30cc03f815884d8f18f35ab47626ccc83
Status: Image is up to date for oracle/graalvm-ce:1.0.0-rc11
Build on Server(pid: 10, port: 32789)*
[hubstats.core:10]    classlist:   2,757.55 ms
[hubstats.core:10]        (cap):     627.82 ms
[hubstats.core:10]        setup:   1,675.92 ms

ls
[hubstats.core:10]     analysis: 484,784.87 ms
11 fatal errors detected:
Exception in thread "ForkJoinPool-3-worker-5" fatal error: java.lang.OutOfMemoryError: GC overhead limit exceeded

I had forgotten that all the important code was commented out. :sob:

nicokosi commented 5 years ago

I am now able to report unsupported features while generating native image :feelsgood: :

Error: unsupported features in 5 methods
Detailed message:
Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image run time. The object was probably created by a class initializer and is reachable from a static field. By default, all class initialization is done during native image building.You can manually delay class initialization to image run time by using the option --delay-class-initialization-to-runtime=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace:  object clojure.lang.Var
    object java.lang.Object[]
    object clojure.lang.PersistentHashMap$BitmapIndexedNode
    object clojure.lang.PersistentHashMap$INode[]
    object clojure.lang.PersistentHashMap$ArrayNode
    object clojure.lang.PersistentHashMap$INode[]
    object clojure.lang.PersistentHashMap$ArrayNode
    object clojure.lang.PersistentHashMap
    object java.util.concurrent.atomic.AtomicReference
    object clojure.lang.Namespace
    object java.lang.Object[]
    object clojure.lang.PersistentHashMap$BitmapIndexedNode
    object clojure.lang.PersistentHashMap
    object java.util.concurrent.atomic.AtomicReference
    object clojure.lang.Namespace
    object clojure.lang.Var
    method hubstats.core$_main.invokeStatic(ISeq)
Call path from entry point to hubstats.core$_main.invokeStatic(ISeq): 
    at hubstats.core$_main.invokeStatic(core.clj:47)
    at hubstats.core$_main.doInvoke(core.clj:47)
    at clojure.lang.RestFn.applyTo(RestFn.java:137)
    at hubstats.core.main(Unknown Source)
    at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
    at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: Detected a started Thread in the image heap. Threads running in the image generator are no longer running at image run time. The object was probably created by a class initializer and is reachable from a static field. By default, all class initialization is done during native image building.You can manually delay class initialization to image run time by using the option --delay-class-initialization-to-runtime=<class-name>. Or you can write your own initialization methods and call them explicitly from your main entry point.
Trace:  object java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
    object java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
    object java.util.concurrent.DelayQueue
    object clojure.lang.Var
    method clojure.core.async.impl.timers$timeout.invokeStatic(long)
Call path from entry point to clojure.core.async.impl.timers$timeout.invokeStatic(long): 
    at clojure.core.async.impl.timers$timeout.invokeStatic(timers.clj:43)
    at clojure.core.async.impl.timers$timeout.invoke(timers.clj:43)
    at clojure.lang.AFn.applyToHelper(AFn.java:154)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at hubstats.core.main(Unknown Source)
    at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
    at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime: javax.net.ssl.SSLContext
Trace:  object java.lang.Object[]
    object clojure.lang.PersistentArrayMap
    object clojure.lang.Var
    object java.lang.Object[]
    object clojure.lang.PersistentHashMap$BitmapIndexedNode
    object clojure.lang.PersistentHashMap$INode[]
    object clojure.lang.PersistentHashMap$ArrayNode
    object clojure.lang.PersistentHashMap$INode[]
    object clojure.lang.PersistentHashMap$ArrayNode
    object clojure.lang.PersistentHashMap
    object java.util.concurrent.atomic.AtomicReference
    object clojure.lang.Namespace
    object java.lang.Object[]
    object clojure.lang.PersistentArrayMap
    object java.util.concurrent.atomic.AtomicReference
    object clojure.lang.Namespace
    object clojure.lang.Var
    method clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object)
Call path from entry point to clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object): 
    at clj_http.client$wrap_accept$fn__3338.invoke(client.clj:722)
    at clojure.lang.AFn.applyToHelper(AFn.java:160)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at hubstats.core.main(Unknown Source)
    at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
    at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: No instances are allowed in the image heap for a class that is initialized or reinitialized at image runtime: sun.security.provider.NativePRNG
Trace:  object java.security.SecureRandom
    object sun.security.ssl.SSLContextImpl$TLSContext
    object sun.security.ssl.SSLSocketFactoryImpl
    object org.apache.http.conn.ssl.SSLConnectionSocketFactory
    object clojure.lang.Var
    object java.lang.Object[]
    object clojure.lang.PersistentHashMap$BitmapIndexedNode
    object clojure.lang.PersistentHashMap$INode[]
    object clojure.lang.PersistentHashMap$ArrayNode
    object clojure.lang.PersistentHashMap$INode[]
    object clojure.lang.PersistentHashMap$ArrayNode
    object clojure.lang.PersistentHashMap
    object java.util.concurrent.atomic.AtomicReference
    object clojure.lang.Namespace
    object java.lang.Object[]
    object clojure.lang.PersistentArrayMap
    object java.util.concurrent.atomic.AtomicReference
    object clojure.lang.Namespace
    object clojure.lang.Var
    method clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object)
Call path from entry point to clj_http.client$wrap_accept$fn__3338.invoke(Object, Object, Object): 
    at clj_http.client$wrap_accept$fn__3338.invoke(client.clj:722)
    at clojure.lang.AFn.applyToHelper(AFn.java:160)
    at clojure.lang.AFn.applyTo(AFn.java:144)
    at hubstats.core.main(Unknown Source)
    at com.oracle.svm.core.JavaMainWrapper.run(JavaMainWrapper.java:152)
    at com.oracle.svm.core.code.IsolateEnterStub.JavaMainWrapper_run_5087f5482cc9a6abc971913ece43acb471d2631b(generated:0)
Error: unbalanced monitors: mismatch at monitorexit, 96|LoadField#lockee__5436__auto__ != 3|LoadField#lockee__5436__auto__
Call path from entry point to clojure.spec.gen.alpha$dynaload$fn__2628.invoke(): 
    at clojure.spec.gen.alpha$dynaload$fn__2628.invoke(alpha.clj:21)
    at clojure.lang.AFn.run(AFn.java:22)
    at java.lang.Thread.run(Thread.java:748)
    at com.oracle.svm.core.thread.JavaThreads.threadStartRoutine(JavaThreads.java:473)
    at com.oracle.svm.core.posix.thread.PosixJavaThreads.pthreadStartRoutine(PosixJavaThreads.java:193)
    at com.oracle.svm.core.code.IsolateEnterStub.PosixJavaThreads_pthreadStartRoutine_e1f4a8c0039f8337338252cd8734f63a79b5e3df(generated:0)
Original exception that caused the problem: org.graalvm.compiler.code.SourceStackTraceBailoutException$1: unbalanced monitors: mismatch at monitorexit, 96|LoadField#lockee__5436__auto__ != 3|LoadField#lockee__5436__auto__
    at clojure.spec.gen.alpha$dynaload$fn__2628.invoke(alpha.clj:22)
Caused by: org.graalvm.compiler.core.common.PermanentBailoutException: unbalanced monitors: mismatch at monitorexit, 96|LoadField#lockee__5436__auto__ != 3|LoadField#lockee__5436__auto__
    at org.graalvm.compiler.java.BytecodeParser.bailout(BytecodeParser.java:3673)
    at org.graalvm.compiler.java.BytecodeParser.genMonitorExit(BytecodeParser.java:2654)
    at org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5031)
    at org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3184)
    at org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:2993)
    at org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:891)
    at org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:785)
    at org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:95)
    at org.graalvm.compiler.phases.Phase.run(Phase.java:49)
    at org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:197)
    at org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
    at org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
    at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:213)
    at com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:332)
    at com.oracle.graal.pointsto.flow.MethodTypeFlow.doParse(MethodTypeFlow.java:310)
    at com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureParsed(MethodTypeFlow.java:300)
    at com.oracle.graal.pointsto.flow.MethodTypeFlow.addContext(MethodTypeFlow.java:107)
    at com.oracle.graal.pointsto.flow.SpecialInvokeTypeFlow.onObservedUpdate(InvokeTypeFlow.java:421)
    at com.oracle.graal.pointsto.flow.TypeFlow.notifyObservers(TypeFlow.java:352)
    at com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:394)
    at com.oracle.graal.pointsto.flow.SourceTypeFlowBase.update(SourceTypeFlowBase.java:121)
    at com.oracle.graal.pointsto.BigBang$2.run(BigBang.java:508)
    at com.oracle.graal.pointsto.util.CompletionExecutor.lambda$execute$0(CompletionExecutor.java:169)
    at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
nicokosi commented 5 years ago

Attempt #7897123 😜

With graalvm-ce:19.0.0: 

./generate-executable.sh

# ✂ snip!
Warning: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Build on Server(pid: 167, port: 42445)*
[hubstats.core:167]    classlist:  10,654.27 ms
[hubstats.core:167]        (cap):   1,121.64 ms
[hubstats.core:167]        setup:   2,439.03 ms
[hubstats.core:167]   (typeflow):   3,412.47 ms
[hubstats.core:167]    (objects):     945.15 ms
[hubstats.core:167]   (features):     265.40 ms
[hubstats.core:167]     analysis:   4,722.17 ms
[hubstats.core:167]     (clinit):      96.55 ms
[hubstats.core:167]     universe:     410.29 ms
[hubstats.core:167]      (parse):     624.01 ms
[hubstats.core:167]     (inline):   1,425.12 ms
[hubstats.core:167]    (compile):   4,951.31 ms
[hubstats.core:167]      compile:   7,264.01 ms
[hubstats.core:167]        image:     408.76 ms
[hubstats.core:167]        write:      97.27 ms
[hubstats.core:167]      [total]:  26,167.22 ms
Warning: Image 'hubstats.core' is a fallback image that requires a JDK for execution (use --no-fallback to suppress fallback image generation).
 => Check the executable:
Display statistics for GitHub pull requests.
Mandatory parameters:
    --organization      GitHub organization
    -o          GitHub organization (shorthand)
    --repository        GitHub repository
    -r          GitHub repository (shorthand)
    --token         GitHub access token (optional)
    -t          GitHub access token (shorthand, optional)
Optional parameters:
    --repositories      Comma-separated list of repositories (optional)
    --since-weeks       output events that occcured since this number of weeks (optional, default: 1)
    -w          output events that occcured since this number of weeks (shorthand, optional, default: 1)
    --since-days        output events that occcured since this number of days (optional)
    -d          output events that occcured since this number of days (shorthand, optional)
    --since         output events that occcured since a date with format '"yyyy-MM-dd'T'HH:mm:ssZ' (optional)
    -s          output events that occcured since a date with format '"yyyy-MM-dd'T'HH:mm:ssZ' (shorthand, optional)

Examples:
    lein run --organization docker --repository containerd --token $token
    lein run --organization docker --repository containerd --since "2017-01-17T00:00:00Z"
    lein run --organization docker --repository containerd --since-days 10
    lein run --organization docker --repositories docker,containerd
 => Copy it to current directory:
nicokosi commented 5 years ago

Merging this pull request with the fallback mode. I'll see later how to fix native-imagecompilation errors and use the --no-fallback mode.