How to compile VM suite?

[MartyIX]

Hi,

I have tried to compile VM suite according to https://github.com/oracle/graal/tree/master/vm#base-graalvm and this is what I get:

marty@marty-GB-BXi7-5500:~/work/programs/graal-projects/graal/vm$ mx --dy /substratevm,/tools,sulong,/graal-nodejs build
Cloning https://github.com/oracle/js.git revision 83b9324bf510ccc9a1a12b35900630fe29079a67 to /home/marty/work/programs/graal-projects/js with Git
Cloning https://github.com/oracle/js.git to /home/marty/work/programs/graal-projects/js with Git
Cloning into '/home/marty/work/programs/graal-projects/js'...
Username for 'https://github.com': ^C

so it seems that https://github.com/oracle/js.git is private/old repository. Is that correct?

My next attempt was to remove /graal-nodejs, then I get:

marty@marty-GB-BXi7-5500:~/work/programs/graal-projects/graal/vm$ mx --dy /substratevm,/tools,sulong build
build: Checking SubstrateVM requirements for building ...
Compiling org.graalvm.component.installer.test with javac-daemon... [/home/marty/work/programs/graal-projects/graal/vm/mxbuild/src/org.graalvm.component.installer.test/bin/org/graalvm/component/installer/ChunkedConnection.class does not exist]
/home/marty/work/programs/graal-projects/graal/vm/src/org.graalvm.component.installer.test/src/org/graalvm/component/installer/persist/FileDownloaderTest.java:44: error: package sun.net does not exist
import sun.net.ConnectionResetException;
              ^
/home/marty/work/programs/graal-projects/graal/vm/src/org.graalvm.component.installer.test/src/org/graalvm/component/installer/persist/FileDownloaderTest.java:259: error: cannot find symbol
        conn.readException = new ConnectionResetException();
                                 ^
  symbol:   class ConnectionResetException
  location: class FileDownloaderTest
2 errors

Compiling org.graalvm.component.installer.test with javac-daemon failed
1 build tasks failed

Am I doing something wrong?

Regards, Martin

[neomatrix369]

@gilles-duboscq I have beefed up the MacOS system, lets see if we get a binary out of it. In the meanwhile we have made some progress https://github.com/neomatrix369/awesome-graal/blob/247a7ae88e59e3e2c8d0114dd24770b672e36add/build/x86_64/linux_macos/docker-build.logs (we are building these components from the suite: GRAALVM_SUITE_RUNTIMES=/substratevm,/tools,sulong,/graal-nodejs,/fastr,truffleruby,graalpython)

Which logs or where can I find logs for builds steps: Building lli.image... and subsequent Building actions?

Would be nice if there's error handling in place at various stages pointing out to what went wrong and which log files to look at.

[gilles-duboscq]

In the meanwhile we have made some progress

Glad to hear that :)

For Building lli.image, there is no extra logging, it will be inline on stderr & stdout. The log seems to be abruptly truncated, is stderr not captured for some reason? In particular if that step fails it should at least print some message saying so to stderr.

[neomatrix369]

@gilles-duboscq

Both stdout and stderr are piped into the log file, so it if errors don't show up there then its worrying. What command leads to running this building image stage (lli.image, graalpython.image, graaljs.image, etc..) ? I would like to run this command separately to debug why it's failing on Linux without giving much information.

Btw we have a successful build on MacOS (including FastR building), see https://raw.githubusercontent.com/neomatrix369/awesome-graal/8e77d36e537d42a62b3570a1ae74b0a40ab99c77/build/x86_64/linux_macos/macos-build.logs, one question there, which folders across the different packages have the runtime binaries - the ones we would like to pack into the final JDK image. Is this done manually or via some command?

I also found these folders, do they contain artefacts that can be bundled with the graalvm suite?:

/path/to/graalvm-repos/fastr/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/fastr/mxbuild/dists
/path/to/graalvm-repos/graal/truffle/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/graal/truffle/mxbuild/dists
/path/to/graalvm-repos/graal/substratevm/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/graal/substratevm/mxbuild/dists
/path/to/graalvm-repos/graal/compiler/mxbuild/dists
/path/to/graalvm-repos/graal/tools/mxbuild/dists
/path/to/graalvm-repos/graal/vm/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/graal/vm/mxbuild/dists
/path/to/graalvm-repos/graal/sdk/mxbuild/dists
/path/to/graalvm-repos/graal/regex/mxbuild/dists
/path/to/graalvm-repos/truffleruby/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/truffleruby/mxbuild/dists
/path/to/graalvm-repos/graalpython/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/graalpython/mxbuild/dists
/path/to/graalvm-repos/sulong/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/sulong/mxbuild/dists
/path/to/graalvm-repos/graal-jvmci-8/mxbuild/dists
/path/to/graalvm-repos/graaljs/graal-nodejs/mxbuild/linux-amd64/dists
/path/to/graalvm-repos/graaljs/graal-nodejs/mxbuild/dists
/path/to/graalvm-repos/graaljs/graal-js/mxbuild/dists

[gilles-duboscq]

These images are built as part of the mx build command that your run in the vm suite.

After the build, the final image exists in 2 place: one directory with all the files and a tar. If you built with mx ${OPTIONS} build, then you can use mx ${OPTIONS} graalvm-home to find the directory. The tar is in mxbuild/<platform>/dists/$(mx ${OPTIONS} graalvm-dist-name | tr '[:upper:]' '[:lower:]' | tr _ -).tar.

[neomatrix369]

@gilles-duboscq Thanks that will help me create the archive for MacOS. For the linux/docker error I think I know what's happening, some OOME killer is sending signal 9 (KILL) to the mx process which is ending up exiting with an error code 137.

I'm investigating how to eliminate OOME, but it would be great if possible on your end to trap the important signals SIGTERM, SIGHUP, SIGKILL, SIGQUIT, etc... and handle the kills coming from outside and gracefully close the process giving some meaningful errors. So we know where this has occurred (you might already be doing all of this, in that case pls ignore me).

[neomatrix369]

@gilles-duboscq with regards to the tar you mention in https://github.com/oracle/graal/issues/415#issuecomment-409907298, is this tar containing the whole of the JDK with graal, and the runtimes we built with prior commands i.e. the GraalVM suite like the one we download from OTN? And ready to be distributed to end-users?

[gilles-duboscq]

mx already handles signals like SIGINT, SIGTERM and SIGQUIT but SIGKILL can not be handled.

Yes the tar is the full GraalVM, including the JDK.