oracle / graal

GraalVM compiles Java applications into native executables that start instantly, scale fast, and use fewer compute resources 🚀
https://www.graalvm.org
Other
20.19k stars 1.62k forks source link

graal/compiler build failure unsupported option '-print-multi-os-directory' #3204

Closed yingsu00 closed 3 years ago

yingsu00 commented 3 years ago

Describe the issue mx build for graal/compiler failed on MacOS

Steps to reproduce the issue git clone git@github.com:yingsu00/graal.git git clone git@github.com:graalvm/mx.git export PATH=$PWD/mx:$PATH cd ~/graal/compiler mx build

Describe GraalVM and your environment:

More details Note that all the files in /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods are actually there but the mx build was unable to read them.

$ mx build
JAVA_HOME: /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home
EXTRA_JAVA_HOMES: /Library/Java/JavaVirtualMachines/jdk-11.0.5.jdk/Contents/Home
13 unsatisfied dependencies were removed from build (use -v to list them)
1 non-default dependencies were removed from build (use -v to list them, mx build --all to build them)
Compiling com.oracle.mxtool.compilerserver with javac(JDK 11)... [/Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.compilerserver/bin/com/oracle/mxtool/compilerserver/ECJDaemon.class does not exist]
Compiling org.graalvm.compiler.nodeinfo with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.nodeinfo/bin/org/graalvm/compiler/nodeinfo/NodeCycles.class does not exist]
Compiling org.graalvm.word with javac-daemon(JDK 11)... [/Users/yingsu/graal/sdk/mxbuild/src/org.graalvm.word/bin/org/graalvm/word/Pointer.class does not exist]
Compiling org.graalvm.graphio with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.graphio/bin/org/graalvm/graphio/GraphProtocol.class does not exist]
Compiling com.oracle.truffle.api.jdk11 with javac-daemon(JDK 11)... [/Users/yingsu/graal/truffle/mxbuild/src/com.oracle.truffle.api.jdk11/bin/com/oracle/truffle/api/impl/TruffleJDKServices.class does not exist]
Compiling org.graalvm.libgraal.jni.annotation with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.libgraal.jni.annotation/bin/org/graalvm/libgraal/jni/annotation/JNIFromLibGraalRepeated.class does not exist]
Building polyglot.image-bash... [/Users/yingsu/graal/sdk/mxbuild/darwin-amd64/polyglot.image-bash/polyglot does not exist]
WARNING: Found multiple locations for 'dependency:sdk:LAUNCHER_COMMON' in 'GRAALVM_3BF9265B58_JAVA11': ['graalvm-3bf9265b58-java11-21.1.0-dev/Contents/Home/lib/graalvm/launcher-common.jar', 'graalvm-3bf9265b58-java11-21.1.0-dev/Contents/Home/lib/graalvm/launcher-common.jar']
Compiling com.oracle.mxtool.junit.jdk9 with javac-daemon(JDK 11)... [/Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.junit.jdk9/bin/com/oracle/mxtool/junit/ModuleSupport.class does not exist]
Compiling org.graalvm.compiler.api.directives with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.api.directives/bin/org/graalvm/compiler/api/directives/GraalDirectives.class does not exist]
Compiling org.graalvm.collections with javac-daemon(JDK 11)... [/Users/yingsu/graal/sdk/mxbuild/src/org.graalvm.collections/bin/org/graalvm/collections/UnmodifiableEconomicMap.class does not exist]
Archiving TRUFFLE_NFI_GRAALVM_SUPPORT... [/Users/yingsu/graal/truffle/mxbuild/dists/truffle-nfi-graalvm-support.tar does not exist]
Compiling com.oracle.mxtool.webserver with javac-daemon(JDK 11)... [/Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.webserver/bin/com/oracle/mxtool/webserver/WebServer.class does not exist]
Compiling org.graalvm.compiler.processor with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.processor/bin/org/graalvm/compiler/processor/AbstractProcessor.class does not exist]
Compiling com.oracle.mxtool.checkcopy with javac-daemon(JDK 11)... [/Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.checkcopy/bin/com/oracle/mxtool/checkcopy/CheckCopyright.class does not exist]
Compiling com.oracle.graalvm.locator.jdk11 with javac-daemon(JDK 11)... [/Users/yingsu/graal/truffle/mxbuild/src/com.oracle.graalvm.locator.jdk11/bin/com/oracle/graalvm/locator/JDKServices.class does not exist]
Archiving TRUFFLE_GRAALVM_SUPPORT... [/Users/yingsu/graal/truffle/mxbuild/dists/truffle-graalvm-support.tar does not exist]
Compiling org.graalvm.options with javac-daemon(JDK 11)... [/Users/yingsu/graal/sdk/mxbuild/src/org.graalvm.options/bin/org/graalvm/options/OptionValues.class does not exist]
Compiling org.graalvm.compiler.truffle.runtime.serviceprovider with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.truffle.runtime.serviceprovider/bin/org/graalvm/compiler/truffle/runtime/serviceprovider/TruffleRuntimeServices.class does not exist]
Compiling com.oracle.truffle.dsl.processor with javac-daemon(JDK 11)... [/Users/yingsu/graal/truffle/mxbuild/src/com.oracle.truffle.dsl.processor/bin/META-INF/services/javax.annotation.processing.Processor does not exist]
Compiling org.graalvm.compiler.truffle.common.hotspot.libgraal with javac-daemon(JDK 11)... [dependency org.graalvm.libgraal.jni.annotation updated]
Compiling org.graalvm.compiler.hotspot.management.libgraal.annotation with javac-daemon(JDK 11)... [dependency org.graalvm.libgraal.jni.annotation updated]
Compiling org.graalvm.compiler.api.runtime with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.api.runtime/bin/org/graalvm/compiler/api/runtime/GraalJVMCICompiler.class does not exist]
Compiling org.graalvm.compiler.serviceprovider with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.serviceprovider/bin/org/graalvm/compiler/serviceprovider/GraalUnsafeAccess.class does not exist]
Compiling org.graalvm.compiler.api.replacements with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.api.replacements/bin/org/graalvm/compiler/api/replacements/ClassSubstitution.class does not exist]
Compiling org.graalvm.compiler.options.processor with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.processor updated]
Compiling org.graalvm.compiler.nodeinfo.processor with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.processor updated]
Compiling org.graalvm.compiler.core.match.processor with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.processor updated]
Compiling org.graalvm.libgraal.jni.processor with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.processor updated]
Compiling org.graalvm.compiler.bytecode with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.compiler.bytecode/bin/org/graalvm/compiler/bytecode/Bytecodes.class does not exist]
Compiling org.graalvm.compiler.serviceprovider.processor with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.processor updated]
Archiving LLVM_TOOLCHAIN... [/Users/yingsu/graal/sdk/mxbuild/darwin-amd64/dists/llvm-toolchain.tar does not exist]
Compiling org.graalvm.micro.benchmarks with javac-daemon(JDK 11)... [/Users/yingsu/graal/compiler/mxbuild/src/org.graalvm.micro.benchmarks/bin/micro/benchmarks/ArrayListBenchmark.class does not exist]
Compiling org.graalvm.collections.test with javac-daemon(JDK 11)... [dependency org.graalvm.collections updated]
Compiling org.graalvm.compiler.api.test with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.api.runtime updated]
Compiling org.graalvm.compiler.truffle.runtime.serviceprovider.jdk11 with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.truffle.runtime.serviceprovider updated]
Compiling org.graalvm.nativeimage with javac-daemon(JDK 11)... [dependency org.graalvm.word updated]
Compiling org.graalvm.compiler.serviceprovider.jdk11 with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.serviceprovider updated]
Compiling org.graalvm.compiler.replacements.processor with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.processor updated]
Compiling com.oracle.mxtool.junit with javac-daemon(JDK 11)... [/Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.junit/bin/com/oracle/mxtool/junit/TestResultLoggerDecorator.class does not exist]
Archiving GRAAL_GRAPHIO... [dependency org.graalvm.graphio updated]
Compiling org.graalvm.compiler.truffle.compiler.hotspot.libgraal.processor with javac-daemon(JDK 11)... [dependency org.graalvm.libgraal.jni.processor updated]
Compiling com.oracle.mxtool.jmh_1_21 with javac-daemon(JDK 11)... [/Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.jmh_1_21/bin/com/oracle/mxtool/jmh_1_21/TestJMH_1_21.class does not exist]
Compiling org.graalvm.compiler.truffle.common with javac-daemon(JDK 11)... [dependency org.graalvm.graphio updated]
Compiling org.graalvm.compiler.hotspot.management.libgraal.processor with javac-daemon(JDK 11)... [dependency org.graalvm.compiler.hotspot.management.libgraal.annotation updated]
Building libffi...
Extracting LIBFFI_SOURCES...
Archiving MX_MICRO_BENCHMARKS... [dependency com.oracle.mxtool.jmh_1_21 updated]
Archiving JUNIT_TOOL... [dependency com.oracle.mxtool.junit.jdk9 updated]
Archiving GRAAL_PROCESSOR... [dependency org.graalvm.compiler.processor updated]
Compiling com.oracle.mxtool.jacoco with javac-daemon(JDK 11)... [/Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.jacoco/bin/com/oracle/mxtool/jacoco/JacocoReport.class does not exist]
Compiling org.graalvm.home with javac-daemon(JDK 11)... [dependency org.graalvm.nativeimage updated]
Compiling org.graalvm.nativeimage.test with javac-daemon(JDK 11)... [dependency org.graalvm.nativeimage updated]
Applying patches...
Building libffi with GNU Make...
CONFIGURE libffi
Compiling org.graalvm.compiler.management with javac-daemon(JDK 11)... [dependency GRAAL_PROCESSOR updated]
Archiving GRAAL_LIBGRAAL_PROCESSOR... [dependency org.graalvm.libgraal.jni.processor updated]
Compiling org.graalvm.polyglot with javac-daemon(JDK 11)... [dependency org.graalvm.collections updated]
Archiving MX_JACOCO_REPORT... [dependency com.oracle.mxtool.jacoco updated]
Compiling org.graalvm.compiler.truffle.common.hotspot with javac-daemon(JDK 11)... [dependency GRAAL_PROCESSOR updated]
WARNING: /Users/yingsu/mx/mxbuild/dists/jdk1.8/mx-jacoco-report.jar: avoid overwrite of about.html
  new: /Users/yingsu/.mx/cache/JACOCOREPORT_0.8.4_e5ca9511493b7e3bc2cabdb8ded92e855f3aac32/jacocoreport-0.8.4.jar!about.html
  old: /Users/yingsu/.mx/cache/JACOCOCORE_0.8.4_53addc878614171ff0fcbc8f78aed12175c22cdb/jacococore-0.8.4.jar!about.html
WARNING: /Users/yingsu/mx/mxbuild/dists/jdk1.8/mx-jacoco-report.jar: avoid overwrite of about.html
  new: /Users/yingsu/.mx/cache/JACOCOREPORT_0.8.4_e5ca9511493b7e3bc2cabdb8ded92e855f3aac32/jacocoreport-0.8.4.jar!about.html
  old: /Users/yingsu/.mx/cache/JACOCOCORE_0.8.4_53addc878614171ff0fcbc8f78aed12175c22cdb/jacococore-0.8.4.jar!about.html
Compiling org.graalvm.home.test with javac-daemon(JDK 11)... [dependency org.graalvm.home updated]
Archiving GRAAL_SDK... [dependency org.graalvm.polyglot updated]
Compiling org.graalvm.launcher with javac-daemon(JDK 11)... [dependency org.graalvm.polyglot updated]
Compiling org.graalvm.polyglot.tck with javac-daemon(JDK 11)... [dependency org.graalvm.polyglot updated]
Building Java module org.graalvm.sdk (graal-sdk.jar) from GRAAL_SDK
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jartool.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jdeps.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jstatd.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jdwp.agent.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.pack.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.smartcardio.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.hotspot.agent.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jfr.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.rmic.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.crypto.cryptoki.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.management.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.rmi.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.net.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.prefs.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.base.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jlink.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.scripting.nashorn.shell.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jcmd.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.desktop.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.attach.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jshell.jmod
Compiling org.graalvm.launcher.test with javac-daemon(JDK 11)... [dependency org.graalvm.launcher updated]
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.management.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.compiler.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.instrument.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.management.agent.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.security.auth.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.scripting.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jdi.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.crypto.ec.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.jconsole.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/jdk.javadoc.jmod
error: cannot read file /Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/jmods/java.security.jgss.jmod
32 errors

rm: conftest.dSYM: is a directory
rm: conftest.dSYM: is a directory
rm: conftest.dSYM: is a directory
rm: conftest.dSYM: is a directory
clang: error: unsupported option '-print-multi-os-directory'
clang: error: no input files
MAKE libffi
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .libs/libffi_convenience.a(java_raw_api.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .libs/libffi_convenience.a(java_raw_api.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .libs/libffi.a(java_raw_api.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: .libs/libffi.a(java_raw_api.o) has no symbols
Archiving GRAAL_SDK failed
error while killing subprocess 8886 "/Library/Java/JavaVirtualMachines/graalvm-ce-java11-21.0.0/Contents/Home/bin/java -cp /Users/yingsu/mx/mxbuild/java/com.oracle.mxtool.compilerserver/bin com.oracle.mxtool.compilerserver.JavacDaemon -j 16": [Errno 3] No such process
1 build tasks failed
chrisseaton commented 3 years ago

I'm not sure I've ever tried using a standard JVM to build Graal. I think it can be done but it might need some configuration. Most people would use a special JVMCI build of OpenJDK https://github.com/graalvm/labs-openjdk-11/releases.

oraluben commented 3 years ago

Are you building GraalVM with GraalVM? I don't think that's the normal use case. You should get a JVMCI JDK (from the link above) and set JAVA_HOME to it. EXTRA_JAVA_HOMES is not necessary here.

yingsu00 commented 3 years ago

@chrisseaton @oraluben Thank you very much. Using the JVMCI JDK works. Does that mean I need to use the JVMCI JDK for my application in order to define my own intrinsic and run on GraalVM ?

chrisseaton commented 3 years ago

You should use JVMCI JDK to build - at the least just to avoid annoying configuration problems and things.

You can run on a recent standard Java I think - it needs some flags and things and isn't always compatible.

What are you trying to achieve? I'd use a GraalVM to run your application unless you've got some special reason or constraint why you can't. It'll just be an easier path all around.

yingsu00 commented 3 years ago

You should use JVMCI JDK to build - at the least just to avoid annoying configuration problems and things.

You can run on a recent standard Java I think - it needs some flags and things and isn't always compatible.

What are you trying to achieve? I'd use a GraalVM to run your application unless you've got some special reason or constraint why you can't. It'll just be an easier path all around.

@chrisseaton Thanks Chris. I'm trying to build compiler intrinsics that supports AVX instructions and use them in my Java application. What build process and configuration do you recommend?

chrisseaton commented 3 years ago

I'd recommend building using JVMCI, and then running your application using the GraalVM you just built.

yingsu00 commented 3 years ago

@chrisseaton Chris, Thank you for answering my questions. There are a few more questions that I have:

I'd like to call some SIMD intrinsics in my application. For example, I want to call BMI instruction PDEP and PEXT in my Java application. To do this I'm thinking to create a BMI2Node which contains a @NodeIntrinsic public static native BigInteger compute(op, value, mask) that emits the masm code, then in my application I can call BMI2Node.compute(PDEP, value, mask). Is this supposed to work?

I tried to call an existing NodeIntrinsic to test my idea above:

public static void main(String[] args) {
        int[] array = new int[100];
        System.out.println(OffsetAddressNode.address(array, 0));
    }

But this doesn't work:

java: package org.graalvm.compiler.nodes.memory.address is not visible
  (package org.graalvm.compiler.nodes.memory.address is declared in module jdk.internal.vm.compiler, which is not in the module graph)

So I wonder if this is possible at all? If yes what's the right way to do it?

ps. I was following the "Compiler Intrinsics" section in https://www.slideshare.net/ThomasWuerthinger/2015-cgo-graal?from_action=save but the example in this talk is no longer available in GraalTutorial.java. I wonder if you can point me to some examples so that I can call SIMD instructions in the application level? If this is possible it will certainly open a new world for the Java community.

chrisseaton commented 3 years ago

Sorry you've sort of exceeded my working knowledge there without spending a lot of time looking into it. Hopefully someone working on this part of the project will address this issue.

oubidar-Abderrahim commented 3 years ago

I believe @dougxc can either provide some insight or pinpoint someone who can. @dougxc Would you mind taking a look at this?

dougxc commented 3 years ago

Please state the JDK you are using (i.e. what does $JAVA_HOME/bin/java -version print out), as well as the full command line for your test.

dougxc commented 3 years ago

Please re-open once a reproducer can be provided.