AArch64: Support CMake build in cross-building environment

[knn-k]

CMake build is enabled by default for OpenJ9 Java 11 for AArch64 Linux. That does not work as expected in cross-building environment as shown below:

-- The ASM compiler identification is GNU
-- Found assembler: /root/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-gcc
-- Looking for clock_gettime
-- Looking for clock_gettime - found
-- OMR: trace root is '/root/openj9/openj9-openjdk-jdk11/build/linux-aarch64-normal-server-release/vm/runtime'
-- Starting with CMake version 3.5.1
-- Found Git: /usr/bin/git (found version "2.7.4")
-- Configuring incomplete, errors occurred!
See also "/root/openj9/openj9-openjdk-jdk11/build/linux-aarch64-normal-server-release/vm/CMakeFiles/CMakeOutput.log".
/root/openj9/openj9-openjdk-jdk11/closed/OpenJ9.gmk:388: recipe for target '/root/openj9/openj9-openjdk-jdk11/build/linux-aarch64-normal-server-release/vm/cmake.stamp' failed
make[3]: *** [/root/openj9/openj9-openjdk-jdk11/build/linux-aarch64-normal-server-release/vm/cmake.stamp] Error 1
/root/openj9/openj9-openjdk-jdk11/closed/custom/Main.gmk:56: recipe for target 'j9vm-build' failed

[knn-k]

Related PRs: ibmruntimes/openj9-openjdk-jdk11#331, #10240, ibmruntimes/openj9-openjdk-jdk11#332

[knn-k]

Other PRs related to AArch64 cross-build: #8814, #9178

[0xdaryl]

@dnakamura : is supporting cross compilation part of the CMake migration plan? And if not, can it be added to the plan? Cross compile builds are actually fairly common practice in a development workflow, especially for smaller and slower devices that have ARM chips. I imagine the same might be true for RISC-V as well.

[knn-k]

I am seeing the following build failure with openj9.dtfj-gensrc-src in my cross build environment on x86-64 Linux. It was OK earlier this week.

Running OpenJ9 preprocessors with OPENJ9_BUILDSPEC: linux_aarch64_cmprssptrs_cross
make[4]: Entering directory '/root/openj9/openj9-openjdk-jdk11/build/linux-aarch64-normal-server-release/vm'
/usr/bin/make -C /root/openj9/openj9-openjdk-jdk11/openj9/sourcetools -f buildj9tools.mk FREEMARKER_JAR=/root/freemarker.jar
make[5]: Entering directory '/root/openj9/openj9-openjdk-jdk11/openj9/sourcetools'
Building /root/openj9/openj9-openjdk-jdk11/build/linux-aarch64-normal-server-release/support/j9tools/uma.jar
make[3]: *** No rule to make target 'all', needed by 'default'.  Stop.
make[2]: *** [openj9.dtfj-gensrc-src] Error 1
make/Main.gmk:112: recipe for target 'openj9.dtfj-gensrc-src' failed

[knn-k]

The failure above seems to have been introduced by https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/380 . I suppose it is not the problem of CMake or not.

[pshipton]

@keithc-ca fyi the previous comment.

[keithc-ca]

That failure is tracked by #11818. I hope to have a fix today.

[pshipton]

The fixes for https://github.com/eclipse/openj9/issues/11818 are merged.

[knn-k]

Running configure with --with-mixedrefs in an cross-building environment failed with the following error:

configure: error: --with-mixedrefs=[static|dynamic] requires --with-cmake
configure exiting with result code 1

[pshipton]

How important is this (https://github.com/eclipse/openj9/issues/10266#issuecomment-779809526), should it hold up switching to use mixedrefs by default? Once mixedrefs is the default cross-building will have to disable mixedrefs during configure, unless we hard code an exception for aarch64.

[knn-k]

You can still build CR or XL binaries after mixedrefs becomes the default, can't you? If yes, I think it would be acceptable.

It is often much faster to build AArch64 binaries on x86 Linux than to build on Raspberry Pis or on servers like cent7-aarch64-1, depending on the performance of the x86 Linux environment.

[pshipton]

We can still build with --with-mixedrefs=no. For now, I'm going to re-work the code to enable it by default so it doesn't enable for cross compiles.

[knn-k]

Copied from my comment https://github.com/ibmruntimes/openj9-openjdk-jdk11/pull/545#issuecomment-1169387006:

CMake does not work with AArch64 Linux cross builds. See the following errors:

With DDR enabled:

CMake Error at /root/openj9/openj9-openjdk-jdk11/omr/cmake/modules/OmrDDRSupport.cmake:36 (message):
  DDR is not supported when cross-compiling.  Use -DOMR_DDR=OFF to disable
  DDR.

With DDR disabled:

Scanning dependencies of target hookgen
[  3%] Building CXX object runtime/omr/tools/hookgen/CMakeFiles/hookgen.dir/HookGen.cpp.o
[  3%] Building CXX object runtime/omr/tools/hookgen/CMakeFiles/hookgen.dir/main.cpp.o
[  3%] Linking CXX executable ../../../hookgen
[  3%] Built target hookgen
[  3%] Generating ut_j9thr.c, ut_j9thr.h, ut_j9thr.pdat
/lib/ld-linux-aarch64.so.1: No such file or directory

hookgen needs to be built statically for running with qemu.

[knn-k]

hookgen, tracegen, tracemerge, and constgen require static linking. There might be some others. constgen also requires static libraries for libj9thr29 and libomrsig.

[janvrany]

@knn-k Hmm...works for me on RISC-V. When you configure for cross compilation with CMake, what''s the value of OMR_EXE_LAUNCHER in your build/.../vm/CMakeCache.txt?

Here I have:

OMR_EXE_LAUNCHER:STRING=/usr/bin/qemu-riscv64-static;-L;/opt/riscv/sysroot

which allows execution of dynamically linked executables (I can actually run the whole J9 this way)

[knn-k]

@janvrany Thank you for the information.

OMR_EXE_LAUNCHER is defined as follows in my cross build environment. Apparently it does not have -L path option.

OMR_EXE_LAUNCHER:STRING=/usr/bin/qemu-aarch64

[janvrany]

@knn-k

Apparently it does not have -L path option.

This is the "problem", the -L QEMU option tells QEMU where to find system libraries. Something like --sysroot for GCC.

If I remember correctly, @dnakamura implemented some kind of automagic configuration of OMR_EXE_LAUNCHER and it seems to work for me on RISC-V.

[knn-k]

Thank you, @janvrany

Now I can build for AArch64 Linux with CMake, by explicitly specifying the --with-sysroot= option. OMR_EXE_LAUNCHER in vm/CMakeCache.txt looks like:

OMR_EXE_LAUNCHER:STRING=/usr/bin/qemu-aarch64;-L;/root/gcc-linaro-7.5.0-2019.12-x86_64_aarch64-linux-gnu/aarch64-linux-gnu/libc

So the remaining problem is CMake + DDR in cross builds.