How to compile VM suite?

[MartyIX]

Hi,

I have tried to compile VM suite according to https://github.com/oracle/graal/tree/master/vm#base-graalvm and this is what I get:

marty@marty-GB-BXi7-5500:~/work/programs/graal-projects/graal/vm$ mx --dy /substratevm,/tools,sulong,/graal-nodejs build
Cloning https://github.com/oracle/js.git revision 83b9324bf510ccc9a1a12b35900630fe29079a67 to /home/marty/work/programs/graal-projects/js with Git
Cloning https://github.com/oracle/js.git to /home/marty/work/programs/graal-projects/js with Git
Cloning into '/home/marty/work/programs/graal-projects/js'...
Username for 'https://github.com': ^C

so it seems that https://github.com/oracle/js.git is private/old repository. Is that correct?

My next attempt was to remove /graal-nodejs, then I get:

marty@marty-GB-BXi7-5500:~/work/programs/graal-projects/graal/vm$ mx --dy /substratevm,/tools,sulong build
build: Checking SubstrateVM requirements for building ...
Compiling org.graalvm.component.installer.test with javac-daemon... [/home/marty/work/programs/graal-projects/graal/vm/mxbuild/src/org.graalvm.component.installer.test/bin/org/graalvm/component/installer/ChunkedConnection.class does not exist]
/home/marty/work/programs/graal-projects/graal/vm/src/org.graalvm.component.installer.test/src/org/graalvm/component/installer/persist/FileDownloaderTest.java:44: error: package sun.net does not exist
import sun.net.ConnectionResetException;
              ^
/home/marty/work/programs/graal-projects/graal/vm/src/org.graalvm.component.installer.test/src/org/graalvm/component/installer/persist/FileDownloaderTest.java:259: error: cannot find symbol
        conn.readException = new ConnectionResetException();
                                 ^
  symbol:   class ConnectionResetException
  location: class FileDownloaderTest
2 errors

Compiling org.graalvm.component.installer.test with javac-daemon failed
1 build tasks failed

Am I doing something wrong?

Regards, Martin

[gilles-duboscq]

Regarding Graal.js, the URL is wrong, until this is fixed, you can replace https://github.com/oracle/js.git with https://github.com/graalvm/graaljs.git in vm/mx.vm/suite.py:

sed -i 's https://github.com/oracle/js.git https://github.com/graalvm/graaljs.git ' vm/mx.vm/suite.py

The second part about ConnectionResetException looks like an issue with the JDK version used for compilation, can you make sure you have JDK8 available? Do you have JAVA_HOME set in the environment or one of mx's env files (~/.mx/env or <graal-repo>/*/mx.*/env)?

[MartyIX]

@gilles-duboscq Thank you! I'll try.

As for the ConnectionResetException, this is my fault as I use JDK 10 by purpose as I thought that Graal can be compiled under any JDK 8+ and the only reason being that your team does not provide binaries for JDK 9+ to general public at the moment to avoid supporting so many JDKs in this early phase.

[gilles-duboscq]

While it is possible to run the various parts of GraalVM on JDK9+, it is not possible to build a GraalVM based on JDK 9+ at the moment.

The deployment and integration of the GraalVM components into the JDK is very different between JDK8 and JDK9+. In JDK8 is is based on class loaders and trusted locations inside the JDK while on JDK9+ it is based on the module system and requires a lot of command-line arguments (module upgrades, opens, exports etc.).

[zmitrok]

@gilles-duboscq do you need any help from the community to get to a JDK9-based GraalVM or is it already in the works and we just can't see the progress being made?

[gilles-duboscq]

Hi @zmitrok, we are working on JDK9+ based GraalVM builds.

[neomatrix369]

With regards to this issue - https://github.com/oracle/graal/issues/415#issuecomment-388354589 (first issue), I had this for graaljs and graalpython and here are my temporary fixes for both Linux and MacOS, if anyone is interested: https://github.com/neomatrix369/awesome-graal/commit/47d7cd2357f11b04ea851645227fd155be5224ed

[neomatrix369]

My latest attempt to build the VM suite failed on MacOS with this error:

[snipped]
clang++ -c -emit-llvm -I/Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/sulong/projects/com.oracle.truffle.llvm.libraries.bitcode/include -I/Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/sulong/projects/com.oracle.truffle.llvm.libraries.bitcode/libcxxabi -I/Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/sulong/projects/com.oracle.truffle.llvm.libraries.bitcode/libcxx -o bin/stdlib_typeinfo.noopt.bc  -std=c++11 -stdlib=libc++ -DLIBCXXABI_SILENT_TERMINATE -MT bin/stdlib_typeinfo.noopt.bc -MMD -MP -MF deps/stdlib_typeinfo.Td /Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/sulong/projects/com.oracle.truffle.llvm.libraries.bitcode/libcxxabi/stdlib_typeinfo.cpp
tar -czf bin/libsulong-src.tar.gz -C /Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/sulong/projects/com.oracle.truffle.llvm.libraries.bitcode .
opt -o bin/abort.bc bin/abort.noopt.bc -mem2reg -globalopt -simplifycfg -constprop -instcombine -dse -loop-simplify -reassociate -licm -gvn
make: opt: No such file or directory
make: *** [bin/abort.bc] Error 1
make: *** Waiting for unfinished jobs....
rm bin/qsort.noopt.bc bin/clock.noopt.bc bin/pthreads.noopt.bc bin/cxa_virtual.noopt.bc bin/abort_message.noopt.bc bin/memcpy.noopt.bc bin/exit.noopt.bc bin/string.noopt.bc bin/fork.noopt.bc bin/cxa_exception.noopt.bc bin/stdlib_exception.noopt.bc bin/assert.noopt.bc bin/cxa_default_handlers.noopt.bc bin/setjmp.noopt.bc bin/crt0.noopt.bc bin/cxa_handlers.noopt.bc bin/abort.noopt.bc bin/exec.noopt.bc bin/sulong_dispose_context.noopt.bc bin/clone.noopt.bc bin/stdlib_typeinfo.noopt.bc bin/private_typeinfo.noopt.bc bin/stat.noopt.bc bin/cxa_exception_storage.noopt.bc bin/builtin.noopt.bc bin/complex.noopt.bc

Building com.oracle.truffle.llvm.libraries.bitcode with GNU Make failed
error while killing subprocess 25633 "/Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/graal-jvmci-8/jdk1.8.0_172/darwin-amd64/product/Contents/Home/bin/java -d64 -cp /Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/mx/mxbuild/java/com.oracle.mxtool.compilerserver/bin:/Users/swami/git-repos/OpenJDK/graal-stuff/awesome-graal/build/x86_64/linux_macos/graal-jvmci-8/jdk1.8.0_172/darwin-amd64/product/Contents/Home/lib/tools.jar com.oracle.mxtool.compilerserver.JavacDaemon -j 4": [Errno 3] No such process
1 build tasks failed

Find full console logs at https://ci.adoptopenjdk.net/job/jdk8-with-graal_build_x86-64_macos/38/console (this might not work after sometime, when Jenkins rotates the console logs)

Here's my script: https://github.com/neomatrix369/awesome-graal/blob/build-graalvm-suite/build/x86_64/linux_macos/local-build.sh#L96

Any thoughts on what must be going wrong?

[chrisseaton]

Do you have LLVM installed? opt is an LLVM command.

[neomatrix369]

Both my Linux and MacOS machines have LLVM, but none have opt, I did some searching and found out, on the MacOS opt has NOT been provided (among couple other commands).

I managed to find opt on my Linux installed under /usr/lib/llvm-4.0 Which versions of LLVM does MX/Graal build expect for both Linux and MacOS?

[chrisseaton]

This is what I have documented for Ruby, but I know these instructions are enough to build Graal as well.

https://github.com/oracle/truffleruby/blob/master/doc/user/installing-llvm.md

[chrisseaton]

Here's Sulong's documentation as well.

https://github.com/graalvm/sulong#build-dependencies

[neomatrix369]

Thanks @chrisseaton working on it now

[neomatrix369]

I have moved further and installed LLVM 6.0 in my docker environment (Linux), and building GraalVM suite:

Compiling com.oracle.svm.core with javac-daemon(JDK 1.8)... [/home/graal/graal/substratevm/mxbuild/src/com.oracle.svm.core/bin/com/oracle/svm/core/UnsafeAccess.class does not exist]
Compiling com.oracle.truffle.r.parser with javac-daemon(JDK 1.8)... [dependency TRUFFLE_R_PARSER_PROCESSOR updated]
Compiling com.oracle.graal.pointsto with javac-daemon(JDK 1.8)... [/home/graal/graal/substratevm/mxbuild/src/com.oracle.graal.pointsto/bin/com/oracle/graal/pointsto/AnalysisObjectScanner.class does not exist]
/home/graal/fastr/com.oracle.truffle.r.parser/src/com/oracle/truffle/r/parser/ParserGeneration.java:37: error: Parser failed to execute command [/home/graal/graal-jvmci-8/openjdk1.8.0-internal/linux-amd64/product/bin/java, -jar, /home/graal/fastr/libdownloads/antlr-complete-3.5.1.jar, -o, /tmp/antlr2606573355340595633.tmp, R.g]. Return code 1.
public class ParserGeneration {
       ^
  Output:error(170): R.g:808:43: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:54: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:98: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:109: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:43: the .. range operator isn't allowed in parser rules

 error(170): R__.g:808:54: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:98: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:109: the .. range operator isn't allowed in parser rules
  warning(200): R.g:710:12:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:716:12:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:716:12:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''A'..'Z'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''_'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''\uFFFD''\uFFFD'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''a'..'z'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:790:29:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:790:44:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

[ ...snipped... ]

  As a result, alternative(s) 4 were disabled for that input
  error(201): R.g:808:20: The following alternatives can never be matched: 4

  warning(200): R.g:808:123:
  Decision can match input such as "'\uFFFD''\uFFFD'" using multiple alternatives: 4, 5

  As a result, alternative(s) 5 were disabled for that input
  error(201): R.g:808:123: The following alternatives can never be matched: 5

1 error

ICU4J dependency found in /home/graal/.mx/cache/ICU4J_6f06e820cf4c8968bbbaae66ae0b33f6a256b57f.jar
3282 files extracted to /home/graal/graaljs/graal-js/lib/icu4j
Use the following parameters when invoking svm version of js to make ICU4J localization data available for the runtime:
-Dpolyglot.js.intl-402=true -Dcom.ibm.icu.impl.ICUBinary.dataPath=/home/graal/graaljs/graal-js/lib/icu4j/icudt
Compiling com.oracle.truffle.r.parser with javac-daemon(JDK 1.8) failed
1 build tasks failed

What is going wrong here, any thoughts?

One of the R tests are failing, does this test pass usually? GraalVM suite build breaks with this above failure, if someone can pls help.

[neomatrix369]

After excluding fastr from the list, the build proceeds but breaks at:

[snipped]
Building com.oracle.truffle.llvm.tests.nfi with GNU Make... [dependency SULONG_LIBS updated]
rm nfi/basicTest/O0.bc
Building com.oracle.truffle.llvm.tests.debug with GNU Make... [dependency SULONG_LIBS updated]
bootstrap.native-image... [dependency SVM_DRIVER updated]
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.isValidJavaVersion(NativeImageGeneratorRunner.java:160)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.verifyValidJavaVersionAndPlatform(NativeImageGeneratorRunner.java:329)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:181)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:378)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:104)

bootstrap.native-image failed
1 build tasks failed

[neomatrix369]

Just raised an issue on the fastr repo about https://github.com/oracle/graal/issues/415#issuecomment-398156370

[neomatrix369]

I have temporarily patched https://github.com/oracle/graal/issues/415#issuecomment-398682772, and get the below (on MacOS):

[snipped]
bootstrap.native-image... [dependency SVM_DRIVER updated]
   classlist:   4,250.13 ms
       (cap):   3,373.13 ms
       setup:   6,804.33 ms
  (typeflow):  18,375.29 ms
   (objects):  16,658.41 ms
  (features):     339.44 ms
    analysis:  35,699.62 ms
    universe:     739.43 ms
     (parse):   6,004.69 ms
    (inline):   7,649.45 ms
   (compile):  30,258.41 ms
     compile:  45,083.91 ms
       image:   4,941.24 ms
       write:   3,953.64 ms
     [total]: 101,625.51 ms
Checking native-image.properties... [dependency TRUFFLENODE_GRAALVM_SUPPORT updated]
Archiving NATIVE_IMAGE... [dependency bootstrap.native-image updated]
WARNING: Could not find dragonegg, not building test variant "gcc_O0"
Archiving GRAALVM_CMP_GU_GVM_INS_JS_NJS_POLYNATIVE_PRO_PYN_RBY_RGX_SLG_STAGE1_SVM_TFL... [dependency TRUFFLENODE_GRAALVM_SUPPORT updated]
  GRAALVM_CMP_GU_GVM_INS_JS_NJS_POLYNATIVE_PRO_PYN_RBY_RGX_SLG_STAGE1_SVM_TFL:
Could not find any source file for 'file:/path/to/graal-stuff/awesome-graal/build/x86_64/linux_macos/graal-jvmci-8/jdk1.8.0_172/darwin-amd64/product'
Archiving GRAALVM_CMP_GU_GVM_INS_JS_NJS_POLYNATIVE_PRO_PYN_RBY_RGX_SLG_STAGE1_SVM_TFL failed
error while killing subprocess 43770 "/path/to/graal-stuff/awesome-graal/build/x86_64/linux_macos/graal-jvmci-8/jdk1.8.0_172/darwin-amd64/product/Contents/Home/bin/java -d64 -cp /path/to/graal-stuff/awesome-graal/build/x86_64/linux_macos/mx/mxbuild/java/com.oracle.mxtool.compilerserver/bin:/path/to/graal-stuff/awesome-graal/build/x86_64/linux_macos/graal-jvmci-8/jdk1.8.0_172/darwin-amd64/product/Contents/Home/lib/tools.jar com.oracle.mxtool.compilerserver.JavacDaemon -j 4": [Errno 3] No such process
1 build tasks failed

I find this error a bit strange:

Could not find any source file for 'file:/path/to/graal-stuff/awesome-graal/build/x86_64/linux_macos/graal-jvmci-8/jdk1.8.0_172/darwin-amd64/product'

[gilles-duboscq]

This (Could not find any source file) looks like what @ansalond just fixed in mx (graalvm/mx@224dfc3)

[ansalond]

Re.:

[snipped]
Building com.oracle.truffle.llvm.tests.nfi with GNU Make... [dependency SULONG_LIBS updated]
rm nfi/basicTest/O0.bc
Building com.oracle.truffle.llvm.tests.debug with GNU Make... [dependency SULONG_LIBS updated]
bootstrap.native-image... [dependency SVM_DRIVER updated]
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.isValidJavaVersion(NativeImageGeneratorRunner.java:160)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.verifyValidJavaVersionAndPlatform(NativeImageGeneratorRunner.java:329)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:181)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:378)
    at com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:104)

bootstrap.native-image failed
1 build tasks failed

Thanks for the report. We have a fix for it, which will be merged in the next hours.

[neomatrix369]

Consider this issue resolved https://github.com/oracle/graal/issues/415#issuecomment-398838894, thank you, we are moving forward :+1: :heart:

[ansalond]

FYI, we removed SubstrateVM's Java version check: https://github.com/oracle/graal/commit/56cbed9d5bd2ff80d6d111f487bb606c7f16bc3f I'm closing this issue, please re-open it if there are more problems. Thanks for all the feedback!

[neomatrix369]

The GraalVM suite build still fails when FastR is included in the suite, I have raised an issue, see https://github.com/oracle/fastr/issues/17. The solution suggested does not resolve the issue, I think everyone will have this issue when building the GraalVM suite.

I suggest to keep it open, unless someone has come up with a solutions for us in the meantime,

[neomatrix369]

I get the below failure when building the Graal compiler:

[snipped]
Compiling /root/project/graal-jvmci-8/src/share/vm/ci/ciInstance.cpp
{standard input}: Assembler messages:

{standard input}:26317: Warning: end of file not at end of a line; newline inserted

{standard input}:28301: Error: unknown pseudo-op: `.lbb43'

{standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive

g++: internal compiler error: Killed (program cc1plus)

Please submit a full bug report,

with preprocessed source if appropriate.

See <file:///usr/share/doc/gcc-5/README.Bugs> for instructions.

make[4]: *** [ad_x86_64_gen.o] Error 4

make[4]: *** Waiting for unfinished jobs....

make[4]: Leaving directory `/root/project/graal-jvmci-8/build/linux/linux_amd64_compiler2/product'
make[3]: Leaving directory `/root/project/graal-jvmci-8/build/linux/linux_amd64_compiler2/product'
make[3]: *** [the_vm] Error 2

make[2]: *** [product] Error 2

make[2]: Leaving directory `/root/project/graal-jvmci-8/build/linux'
make[1]: *** [generic_build2] Error 2

make[1]: Leaving directory `/root/project/graal-jvmci-8/make'
make: *** [product] Error 2

make: Leaving directory `/root/project/graal-jvmci-8/make'

Building HotSpot[product, server] failed
1 build tasks failed
Exited with code 1

gcc version check
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609

g++ version check
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.9) 5.4.0 20160609

[gilles-duboscq]

Hi @neomatrix369,

It looks like an issue with your toolchain. Apparently it's a common error when one of the processes used by GCC gets killed by the OOM killer because of low memory conditions.

How much RAM & CPUs do you have available? By default it will run as many compilations in parallel as you have CPUs. You can try to lower that parallelism to save memory by using the HOTSPOT_BUILD_JOBS environment variable: setting it to 1 would only run one compilation at a time.

[neomatrix369]

Thanks @gilles-duboscq great feedback, and was very timely. Sorry I wasn't available to look at it immediately. It totally helps me with an issue I'm having on a low-spec Mac laptop where NativeImage building (last stage) takes ages and the system hangs. I can lower the HOTSPOT_BUILD_JOBS for such systems.

As such here's the approach I'm using: HOTSPOT_BUILD_JOBS=<use 50% of available threads>

If I notice there is still leg-room and I can stretch a bit more, I'll switch to: HOTSPOT_BUILD_JOBS=<use 75% of available threads>

Both the approaches help keep some leg-room for the rest of the system to run all other processes.

Any thoughts?

[gilles-duboscq]

HOTSPOT_BUILD_JOBS will help on machine which have a bad CPU threads to RAM ratio during the build of graal-jvmci-8.

If you want a more general solution, you can use mx's --max-cpus=N flag which will work for anything built from mx (it's an mx flag so it should be placed before the build sub-command). (During the graal-jvmci-8 build, mx will default HOTSPOT_BUILD_JOBS to the value of --max-cpus)

[neomatrix369]

Thanks @gilles-duboscq for the above update.

[neomatrix369]

Whilst building the graal compiler I get this, building from master (fc92c740a4165c7cb504f2483381d0570d759c1d) and master of graal-jvmci-8 (9d1008340c390a68386b7c17d88a6a43e7162c76):

Compiling com.oracle.truffle.api.vm with javac-daemon(JDK 1.8)... [dependency TRUFFLE_DSL_PROCESSOR_INTEROP_INTERNAL updated]
/home/graal/graal/compiler/src/org.graalvm.compiler.nodes/src/org/graalvm/compiler/nodes/GraphSpeculationLog.java:42: error: GraphSpeculationLog is not abstract and does not override abstract method lookupSpeculation(JavaConstant) in SpeculationLog
public final class GraphSpeculationLog implements SpeculationLog {
             ^
/home/graal/graal/compiler/src/org.graalvm.compiler.nodes/src/org/graalvm/compiler/nodes/GraphSpeculationLog.java:87: error: speculate(SpeculationReason) in GraphSpeculationLog cannot implement speculate(SpeculationReason) in SpeculationLog
    public JavaConstant speculate(SpeculationReason reason) {
                        ^
  return type JavaConstant is not compatible with Speculation
/home/graal/graal/compiler/src/org.graalvm.compiler.nodes/src/org/graalvm/compiler/nodes/GraphSpeculationLog.java:74: error: incompatible types: Speculation cannot be converted to JavaConstant
                    speculation = log.speculate(reason);
                                               ^
/home/graal/graal/compiler/src/org.graalvm.compiler.nodes/src/org/graalvm/compiler/nodes/GraphSpeculationLog.java:86: error: method does not override or implement a method from a supertype
    @Override
    ^
4 errors

Compiling org.graalvm.compiler.nodes with javac-daemon(JDK 1.8) failed
1 build tasks failed

[dougxc]

Hi @neomatrix369 , please use the jvmci-0.45 tagged version of graal-ivmci-8. Changes after that introduce a backwards incompatible API change to JVMCI. I'm currently integrating the Graal changes that adapt to these API changes and will send a message to graal-dev@openjdk.java.net providing more detail once the Graal changes land on GitHub.

[dougxc]

Please see http://mail.openjdk.java.net/pipermail/graal-dev/2018-July/005443.html

[neomatrix369]

Thanks @dougxc

[neomatrix369]

Just recently I'm getting the below error:

...[snipped]....
Compiling com.oracle.truffle.tools.profiler.test with javac-daemon(JDK 1.8)... [dependency com.oracle.truffle.tools.profiler updated]
Compiling com.oracle.svm.core.graal with javac-daemon(JDK 1.8)... [dependency com.oracle.svm.core.posix updated]
Archiving TRUFFLE_PROFILER_TEST... [dependency com.oracle.truffle.tools.profiler.test updated]
Archiving GRAALPYTHON_UNIT_TESTS... [dependency com.oracle.graal.python.test updated]
/home/graal/graal/substratevm/mxbuild/src/com.oracle.svm.core.graal/src_gen/com/oracle/svm/core/graal/windows/PluginFactory_WindowsCEntryPointSnippets.java:151: error: error while writing WindowsCEntryPointSnippets_runtimeCall_ForeignCallDescriptor_CEntryPointCreateIsolateParameters_int: /home/graal/graal/substratevm/mxbuild/src/com.oracle.svm.core.graal/bin/com/oracle/svm/core/graal/windows/PluginFactory_WindowsCEntryPointSnippets$WindowsCEntryPointSnippets_runtimeCall_ForeignCallDescriptor_CEntryPointCreateIsolateParameters_int.class (File name too long)
    private static final class WindowsCEntryPointSnippets_runtimeCall_ForeignCallDescriptor_CEntryPointCreateIsolateParameters_int extends GeneratedInvocationPlugin {
                         ^
1 error

Compiling com.oracle.svm.core.graal with javac-daemon(JDK 1.8) failed
1 build tasks failed

Building from the suggested tag for JVMCI repo and master of oracle/graal repo

[chrisseaton]

Are you compiling on an encrypted file system, or otherwise in some unusual environment?

[neomatrix369]

In a Docker container on CircleCI, I did go past building the graal compiler, but it seems to hang a lot when building the graalvm suite, here's the full logs https://github.com/neomatrix369/awesome-graal/blob/97c339d7a411e900a71f658cf372d11ff23a7ea6/build/x86_64/linux_macos/docker-build.logs

Its got all the HW, SW and other environment related details in it

[neomatrix369]

And on the MacOS it hangs a lot towards the end when building native images, see bottom of these logs https://github.com/neomatrix369/awesome-graal/blob/d9c9f68755275fb64ea0e7f7483bbe97e7b06b79/build/x86_64/linux_macos/macos-build.logs

I have set the threads to 1 even then it never finishes. Is there a minimum HW/SW resources requirements, maybe it could check and alert the end user about it and not proceed with the build.

[olpaw]

@neomatrix369 on which platform are you building when you get the error you are describing in https://github.com/oracle/graal/issues/415#issuecomment-405043425 ?

[olpaw]

@neomatrix369 regarding minimum HW/SW resources requirements I can say that we only test on machines with at least two cores and >= 16GB RAM. It's fair to say that building native-images without having at least 12GB RAM is likely causing problems (with non-trivial images)

[neomatrix369]

@olpaw Mentioned the details and full log on https://github.com/oracle/graal/issues/415#issuecomment-405049446, platform information can be found from https://github.com/neomatrix369/awesome-graal/blob/97c339d7a411e900a71f658cf372d11ff23a7ea6/build/x86_64/linux_macos/docker-build.logs#L19 onwards

[neomatrix369]

Your comment https://github.com/oracle/graal/issues/415#issuecomment-405309238 will explain why the MacOS build is hanging (mentioned on https://github.com/oracle/graal/issues/415#issuecomment-405049446)

[ansalond]

Looking at your macOS build logs I'm not sure that the build is hanging. I think you had a failure building FastR:

Building com.oracle.truffle.r.native with GNU Make failed

The FastR build in release mode (FASTR_RELEASE=true, required when you build a GraalVM image) is quite slow, mainly due to the internal GNUR build and the installation of recommended packages. To avoid polluting the output, stdout is redirected to <fastr>/libdownloads/R-3.4.0/gnur_make.log. Could you please check the content of this file?

[olpaw]

@neomatrix369 thanks for providing a full log. It appears that you run into a java class-file pathname length limit during javac compilation. The reason this happens with PluginFactory_WindowsCEntryPointSnippets$WindowsCEntryPointSnippets_runtimeCall_ForeignCallDescriptor_CEntryPointCreateIsolateParameters_int.class is because its source file is autogenerated by the graal's annotation processor which usually leads to long filenames. While we did not observe that on our CI infrastructure (we don't build GraalVM in docker) others have observed the same problem when compiling java sources, e.g https://github.com/moby/moby/issues/1413 and https://issues.scala-lang.org/browse/SI-3623?focusedCommentId=74341&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-74341. These sources indicate that the problem is docker related. Maybe we should make mx use something like -Xmax-classfile-name 242 per default. /cc @dougxc @gilles-duboscq

[dougxc]

The problem needs to be addressed by the code generating annotation processors as the mapping from Java source name to Java class file is somewhat fixed.

[dougxc]

I'm pushing a change now that reduces the length of generated class names. For example:

com.oracle.svm.core.graal.windows.PluginFactory_WindowsCEntryPointSnippets$WindowsCEntryPointSnippets_runtimeCall_ForeignCallDescriptor_CEntryPointCreateIsolateParameters_int

is now:

com.oracle.svm.core.graal.windows.PluginFactory_WindowsCEntryPointSnippets.runtimeCall__4

[neomatrix369]

Thanks @ansalond @dougxc @olpaw for your feedback, I'm glad the logs helped fixed things at your end. I'm re-running the builds from scratch to see if I find anything new.

[neomatrix369]

@ansalond in response to https://github.com/oracle/graal/issues/415#issuecomment-405521899, I'm running with FASTR_RELEASE=true will check the GNU R logs and respond

[neomatrix369]

@olpaw for the analysis and @dougxc for the fix, rebuild is in progress, will report when done

[neomatrix369]

@ansalond in response to https://github.com/oracle/graal/issues/415#issuecomment-405521899, - just checked the gnur_make.log file, it contains a single line message: make[3] *** No targets specified and no makefile found....Stop.

I rebuild it on MacOS with fastr enabled in the graalvm suite and got this and after disabling fastr got this. Looks like failing for the same reason.

[neomatrix369]

Building in docker (including the fastr component) fails, see docker build logs

[neomatrix369]

Building in native MacOS (excluding the fastr component) never finishes although nearly at the end, see MacOS build logs

[gilles-duboscq]

Hi @neomatrix369, in the MacOS log there seems that the machine you are using has an i5-2415M with 2 cores × 2 threads with 4GB RAM. It will be very hard to get a build in a reasonable time on this machine for now, especially on the RAM front.

Regarding the other build it seems to be missing some build dependencies for Sulong. In principle libc++-dev should provide those files on ubuntu.

[neomatrix369]

@gilles-duboscq On it now, added it to the list of OS dependencies, and it did make a good difference, I have a different problem to tackle but thats something for me to fix. Thanks again :+1: