Building GraalVM suite with fastr fails due to failing test

[neomatrix369]

In the process of building building GraalVM suite inside a docker environment (Linux), I get the below:

Compiling com.oracle.svm.core with javac-daemon(JDK 1.8)... [/home/graal/graal/substratevm/mxbuild/src/com.oracle.svm.core/bin/com/oracle/svm/core/UnsafeAccess.class does not exist]
Compiling com.oracle.truffle.r.parser with javac-daemon(JDK 1.8)... [dependency TRUFFLE_R_PARSER_PROCESSOR updated]
Compiling com.oracle.graal.pointsto with javac-daemon(JDK 1.8)... [/home/graal/graal/substratevm/mxbuild/src/com.oracle.graal.pointsto/bin/com/oracle/graal/pointsto/AnalysisObjectScanner.class does not exist]
/home/graal/fastr/com.oracle.truffle.r.parser/src/com/oracle/truffle/r/parser/ParserGeneration.java:37: error: Parser failed to execute command [/home/graal/graal-jvmci-8/openjdk1.8.0-internal/linux-amd64/product/bin/java, -jar, /home/graal/fastr/libdownloads/antlr-complete-3.5.1.jar, -o, /tmp/antlr2606573355340595633.tmp, R.g]. Return code 1.
public class ParserGeneration {
       ^
  Output:error(170): R.g:808:43: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:54: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:98: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:109: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:43: the .. range operator isn't allowed in parser rules

 error(170): R__.g:808:54: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:98: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:109: the .. range operator isn't allowed in parser rules
  warning(200): R.g:710:12:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:716:12:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:716:12:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''A'..'Z'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''_'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''\uFFFD''\uFFFD'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5:
  Decision can match input such as "'.''.''.''.''.''a'..'z'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:790:29:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:790:44:
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

[ ...snipped... ]

  As a result, alternative(s) 4 were disabled for that input
  error(201): R.g:808:20: The following alternatives can never be matched: 4

  warning(200): R.g:808:123:
  Decision can match input such as "'\uFFFD''\uFFFD'" using multiple alternatives: 4, 5

  As a result, alternative(s) 5 were disabled for that input
  error(201): R.g:808:123: The following alternatives can never be matched: 5

1 error

ICU4J dependency found in /home/graal/.mx/cache/ICU4J_6f06e820cf4c8968bbbaae66ae0b33f6a256b57f.jar
3282 files extracted to /home/graal/graaljs/graal-js/lib/icu4j
Use the following parameters when invoking svm version of js to make ICU4J localization data available for the runtime:
-Dpolyglot.js.intl-402=true -Dcom.ibm.icu.impl.ICUBinary.dataPath=/home/graal/graaljs/graal-js/lib/icu4j/icudt
Compiling com.oracle.truffle.r.parser with javac-daemon(JDK 1.8) failed
1 build tasks failed

One of the R tests are failing, does this test pass usually?

[neomatrix369]

Scripts that run the above can be found at s https://github.com/neomatrix369/awesome-graal/blob/build-graalvm-suite/build/x86_64/linux_macos/local-build.sh

[steve-s]

Hi Mani,

this looks like ANTLR grammar issue. Can you try exporting LC_ALL=C before building? We've seen this issue occasionally in the past. I will update the documentation.

[neomatrix369]

I have added this to my scripts https://github.com/neomatrix369/awesome-graal/blob/build-graalvm-suite/build/x86_64/linux_macos/local-build.sh#L20 but get the same error, as a result now building by excluding the fastr package temporarily

[zslajchrt]

Hi Mani,

It seems that the grammar parser requires that LANG, LC_ALL and LC_CTYPE environment variables be set to the same value, e.g. en_US.UTF-8.

[neomatrix369]

Thanks @zslajchrt, sorry I was distracted and couldn't apply the above, doing it now, will let you know when its works.

[neomatrix369]

Guys this is still failing:

Compiling com.oracle.truffle.r.parser with javac-daemon(JDK 1.8)... [dependency TRUFFLE_R_PARSER_PROCESSOR updated]
/home/graal/project/fastr/com.oracle.truffle.r.parser/src/com/oracle/truffle/r/parser/ParserGeneration.java:37: error: Parser failed to execute command [/home/graal/project/graal-jvmci-8/openjdk1.8.0-internal/linux-amd64/product/bin/java, -jar, /home/graal/project/fastr/libdownloads/antlr-complete-3.5.1.jar, -o, /tmp/antlr2537488579950569639.tmp, R.g]. Return code 1.
public class ParserGeneration {
       ^
  Output:error(170): R.g:808:43: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:54: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:98: the .. range operator isn't allowed in parser rules
  error(170): R.g:808:109: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:43: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:54: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:98: the .. range operator isn't allowed in parser rules
  error(170): R__.g:808:109: the .. range operator isn't allowed in parser rules
  warning(200): R.g:710:12: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:716:12: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5: 
  Decision can match input such as "'.''.''.''.''.''A'..'Z'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5: 
  Decision can match input such as "'.''.''.''.''.''_'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5: 
  Decision can match input such as "'.''.''.''.''.''\uFFFD''\uFFFD'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:722:5: 
  Decision can match input such as "'.''.''.''.''.''a'..'z'" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:790:29: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:790:44: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:791:29: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:791:44: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:791:59: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:791:74: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:791:89: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:791:104: 
  Decision can match input such as "{'0'..'9', 'A'..'F', 'a'..'f'}" using multiple alternatives: 1, 2

  As a result, alternative(s) 2 were disabled for that input
  warning(200): R.g:808:20: 
  Decision can match input such as "'\uFFFD''\uFFFD'" using multiple alternatives: 3, 4

  As a result, alternative(s) 4 were disabled for that input
  error(201): R.g:808:20: The following alternatives can never be matched: 4

  warning(200): R.g:808:123: 
  Decision can match input such as "'\uFFFD''\uFFFD'" using multiple alternatives: 4, 5

  As a result, alternative(s) 5 were disabled for that input
  error(201): R.g:808:123: The following alternatives can never be matched: 5
1 error

Compiling com.oracle.truffle.js.nodes with javac-daemon(JDK 1.8)... [dependency TRUFFLE_JS_FACTORY_PROCESSOR updated]

Compiling org.truffleruby with javac-daemon(JDK 1.8) failed
Compiling com.oracle.graal.python with javac-daemon(JDK 1.8) failed
Compiling com.oracle.truffle.r.parser with javac-daemon(JDK 1.8) failed
Compiling com.oracle.truffle.js.nodes with javac-daemon(JDK 1.8) failed
4 build tasks failed
Exited with code 1

Here's the env variable settings:

[...snipped...]
export FASTR_RELEASE="true"
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"

[neomatrix369]

I have found out finally how to handle the below settings:

export FASTR_RELEASE="true"
export LANG="en_US.UTF-8"
export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"

we need to install locale and run the below before setting the above env variables:

locale en_US.UTF-8

Although I still get failures with a full suite build, see build logs

[chrisseaton]

This is documented in the Ruby Docker configurations.

https://github.com/oracle/truffleruby/blob/53a4fc71a2d319c31b89804995a6ab3697215ec7/tool/docker-configs.yaml#L88

Did I share this with you earlier?

[neomatrix369]

Actually, I found that out after debugging the error message - thanks for sharing though, helps confirm the solution is correct, and I can use some of the other settings

[steve-s]

Hi Mani,

did you manage to get it to build finally? Some additional errors may be logged in libdownloads/R-3.4.0/gnur_configure.log. There is a docker image (ubuntu based) that includes all the dependencies necessary to build FastR: https://github.com/nuest/fastr-docker.

[neomatrix369]

@steve-s Actually haven;t had a chance to look at it, thanks for the docker image link, should help me get further

[neomatrix369]

Hurray! We have made some progress, see https://github.com/neomatrix369/awesome-graal/blob/247a7ae88e59e3e2c8d0114dd24770b672e36add/build/x86_64/linux_macos/docker-build.logs, pending build results from MacOS, which should come back soon.

Let's keep this issue open until I have a suite for both Linux and MacOS.

[neomatrix369]

FastR builds both on Linux and MacOS, although the finally step does not complete on Linux - investigations underway, thanks all for the help and suggestions so far.

[neomatrix369]

Just for the record installing fortran on MacOS via brew is a bit broken. When trying to locate the dynamic lib file, we need to use a long winded step (its been documented in the local-builds.sh script

[neomatrix369]

Able to build FastR on both Linux and MacOS, although I have found the pcre and fortran on both platforms are not easily found when building the suite.

Although specifying the paths where they exist via the PKG_XXXXX flags does resolve the build issue.