Support Scala3 Graalvm bytecode manipulation for native-image

[carlosedp]

This PR aims to add support to do bytecode manipulation for Scala 3 on GraalVM in the same way as currently done by scala-cli while generating native images.

Fixes #19

[carlosedp]

Until this point, I get:

Exception in thread "MillServerActionRunner" java.lang.NoClassDefFoundError: scala/cli/graal/JarCache
    at io.github.alexarchambault.millnativeimage.NativeImage.$anonfun$nativeImageCsCommand$2(NativeImage.scala:16)
    at mill.define.Task$TraverseCtx.evaluate(Task.scala:380)
    at mill.eval.Evaluator.$anonfun$evaluateGroup$13(Evaluator.scala:627)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
    at scala.Console$.withErr(Console.scala:193)
    at mill.eval.Evaluator.$anonfun$evaluateGroup$12(Evaluator.scala:627)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
    at scala.Console$.withOut(Console.scala:164)
    at mill.eval.Evaluator.$anonfun$evaluateGroup$11(Evaluator.scala:626)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
    at scala.Console$.withIn(Console.scala:227)
    at mill.eval.Evaluator.$anonfun$evaluateGroup$8(Evaluator.scala:625)
    at mill.eval.Evaluator.$anonfun$evaluateGroup$8$adapted(Evaluator.scala:586)
    at scala.collection.immutable.Vector.foreach(Vector.scala:1895)
    at mill.eval.Evaluator.evaluateGroup(Evaluator.scala:586)
    at mill.eval.Evaluator.$anonfun$evaluateGroupCached$21(Evaluator.scala:478)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
    at mill.eval.Evaluator.evaluateGroupCached(Evaluator.scala:469)
    at mill.eval.Evaluator.$anonfun$sequentialEvaluate$2(Evaluator.scala:202)
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:575)
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:573)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1300)
    at mill.eval.Evaluator.sequentialEvaluate(Evaluator.scala:177)
    at mill.eval.Evaluator.evaluate(Evaluator.scala:162)
    at mill.main.RunScript$.evaluateNamed(RunScript.scala:363)
    at mill.main.RunScript$.evaluate(RunScript.scala:349)
    at mill.main.RunScript$.$anonfun$evaluateTasks$1(RunScript.scala:314)
    at scala.util.Either.map(Either.scala:382)
    at mill.main.RunScript$.evaluateTasks(RunScript.scala:312)
    at mill.main.RunScript$.$anonfun$runScript$8(RunScript.scala:105)
    at ammonite.util.Res$Success.flatMap(Res.scala:62)
    at mill.main.RunScript$.runScript(RunScript.scala:104)
    at mill.main.MainRunner.$anonfun$runScript$1(MainRunner.scala:119)
    at mill.main.MainRunner.watchLoop2(MainRunner.scala:67)
    at mill.main.MainRunner.runScript(MainRunner.scala:92)
    at mill.MillMain$.main0(MillMain.scala:310)
    at mill.main.MillServerMain$.main0(MillServerMain.scala:79)
    at mill.main.Server.$anonfun$handleRun$1(MillServerMain.scala:184)
    at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: scala.cli.graal.JarCache
    at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:445)
    at ammonite.runtime.SpecialClassLoader.findClass(ClassLoaders.scala:241)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:587)
    at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)
    ... 39 more

~Maybe I have something missing @romanowski / @alexarchambault.~

Fixed!

[carlosedp]

Fixed, now the native image is generated correctly!

[carlosedp]

Something is missing when running with Docker... it couldn't find the main class in the classpath:

[1/7] Initializing...                                                                                    (0.0s @ 0.13GB)
Error: Main entry point class 'com.carlosedp.zioscalajs.backend.MainApp' neither found on the classpath nor on the modulepath.
modulepath: '/root/.cache/coursier/arc/https/github.com/graalvm/graalvm-ce-builds/releases/download/vm-22.3.0/graalvm-ce-java17-linux-amd64-22.3.0.tar.gz/graalvm-ce-java17-22.3.0/lib/svm/library-support.jar'
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
Error: Image build request failed with exit status 1
1 targets failed
show 1 targets failed
backend.nativeImage os.SubprocessException: CommandResult 1

    os.proc.call(ProcessOps.scala:85)
    io.github.alexarchambault.millnativeimage.NativeImage.$anonfun$nativeImage$9(NativeImage.scala:208)
    mill.define.Task$TraverseCtx.evaluate(Task.scala:380)

Maybe the classpath is getting garbled in the process and fails on Docker. If running locally in the host, it works fine!

[carlosedp]

Thanks for the help @alexarchambault ... I found that the problem with Docker build is that the jars are not being copied over to the working dir since this is done before the BytecodeProcessor.processClassPath call with the "original" classpath items.

[alexarchambault]

@carlosedp And thanks to you for opening this!

I think I'd like to ponder a little the changes here, before merging… mill-native-image badly needs integration tests. For now, the only way to know whether it works or not consists in releasing it, and checking that (actually tested) downstream projects, such as Scala CLI, don't suffer regressions.

[carlosedp]

With latest commits, I'm getting this at my test project:

[1/1] show > [52/53] backend.nativeImageEnableScala3Pre32PostProcessing
1 targets failed
show 1 targets failed
backend.nativeImageEnableScala3Pre32PostProcessing java.nio.file.NoSuchFileException: /Users/cdepaula/repos/scala-playground/zio-scalajs-stack/backend/resources
    java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
    java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
    java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
    java.base/java.nio.file.Files.readAttributes(Files.java:1851)
    java.base/java.util.zip.ZipFile$Source.get(ZipFile.java:1264)
    java.base/java.util.zip.ZipFile$CleanableResource.<init>(ZipFile.java:709)
    java.base/java.util.zip.ZipFile.<init>(ZipFile.java:243)
    java.base/java.util.zip.ZipFile.<init>(ZipFile.java:172)
    java.base/java.util.zip.ZipFile.<init>(ZipFile.java:186)
    io.github.alexarchambault.millnativeimage.NativeImage.$anonfun$nativeImageEnableScala3Pre32PostProcessing$4(NativeImage.scala:43)
    io.github.alexarchambault.millnativeimage.NativeImage.$anonfun$nativeImageEnableScala3Pre32PostProcessing$4$adapted(NativeImage.scala:40)
    scala.collection.IterableOnceOps.exists(IterableOnce.scala:591)
    scala.collection.IterableOnceOps.exists$(IterableOnce.scala:588)
    scala.collection.AbstractIterator.exists(Iterator.scala:1293)
    io.github.alexarchambault.millnativeimage.NativeImage.$anonfun$nativeImageEnableScala3Pre32PostProcessing$2(NativeImage.scala:40)
    mill.define.Task$TraverseCtx.evaluate(Task.scala:380)

[alexarchambault]

Seems I can't use this from the Scala CLI build. Currently getting this upon startup:

error while loading NotGiven, Missing dependency 'Add -Ytasty-reader to scalac options to parse the TASTy in /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/NotGiven.class)', required by /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/NotGiven.class)
error while loading TupledFunction, Missing dependency 'Add -Ytasty-reader to scalac options to parse the TASTy in /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/TupledFunction.class)', required by /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/TupledFunction.class)
error while loading FromDigits, Missing dependency 'Add -Ytasty-reader to scalac options to parse the TASTy in /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/FromDigits.class)', required by /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/FromDigits.class)
error while loading CommandLineParser, Missing dependency 'Add -Ytasty-reader to scalac options to parse the TASTy in /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/CommandLineParser.class)', required by /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/CommandLineParser.class)
error while loading LowPriorityNotGiven, Missing dependency 'Add -Ytasty-reader to scalac options to parse the TASTy in /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/LowPriorityNotGiven.class)', required by /home/alexandre/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala3-library_3/3.2.1/scala3-library_3-3.2.1.jar(scala/util/LowPriorityNotGiven.class)
Compilation Failed

[carlosedp]

Nice, building "locally" worked fine now!

Building on Docker still doesn't work... I get:

[2/7] Performing analysis...  []                                                                        (45.8s @ 0.91GB)
   8,243 (85.86%) of  9,600 classes reachable
  10,730 (57.37%) of 18,703 fields reachable
  39,612 (38.64%) of 102,526 methods reachable
     259 classes,   125 fields, and 1,649 methods registered for reflection

Error: Classes that should be initialized at run time got initialized during image building:
 io.netty.incubator.channel.uring.Native the class was requested to be initialized at run time (from command line with 'io.netty.incubator.channel.uring.Native'). To see why io.netty.incubator.channel.uring.Native got initialized use --trace-class-initialization=io.netty.incubator.channel.uring.Native
To see how the classes got initialized, use --trace-class-initialization=io.netty.incubator.channel.uring.Native
Error: Use -H:+ReportExceptionStackTraces to print stacktrace of underlying exception
------------------------------------------------------------------------------------------------------------------------
                        4.1s (6.9% of total time) in 24 GCs | Peak RSS: 1.96GB | CPU load: 3.08
========================================================================================================================
Failed generating 'output' after 58.7s.
Error: Image build request failed with exit status 1
1 targets failed
show 1 targets failed
backend.nativeImage os.SubprocessException: CommandResult 1

    os.proc.call(ProcessOps.scala:85)
    io.github.alexarchambault.millnativeimage.NativeImage.$anonfun$nativeImage$9(NativeImage.scala:246)
    mill.define.Task$TraverseCtx.evaluate(Task.scala:380)

But I have the opt "--initialize-at-run-time=io.netty.incubator.channel.uring.Native" in my project nativeImageOptions (as it works building locally).

Also, I don't see the modified classes being copied to the docker working dir and added to the classpath on the working-dir/run-native-image.sh script.

[carlosedp]

I think this might not be required anymore with the release of Scala 3.3 that has the correct implementation of LazyVals. What do you think @alexarchambault?

[carlosedp]

Closing