search

oracle / graal

GraalVM compiles Java applications into native executables that start instantly, scale fast, and use fewer compute resources 🚀
https://www.graalvm.org
Other
20.38k stars 1.64k forks source link

[GR-41372] Compiler backend LLVM broken - "ImageSingletons do not contain key com.oracle.svm.core.graal.code.SubstrateLoweringProviderFactory" #4992

Closed shoffmeister closed 2 years ago

shoffmeister commented 2 years ago

Describe the issue

Trying to enable the LLVM compiler backend using -H:CompilerBackend=llvm results in a native image production failure ImageSingletons do not contain key com.oracle.svm.core.graal.code.SubstrateLoweringProviderFactory

The exact same (trivial) project compiles fine without the LLVM backend - i.e.: C compiler: gcc (redhat, x86_64, 12.2.1)

Steps to reproduce the issue

//exp: builds a native image using the LLVM toolchain //act: exception from the Maven plugin

========================================================================================================================
GraalVM Native Image: Generating 'defect-graalvm-llvm' (executable)...
========================================================================================================================
[1/7] Initializing...
                                                                                    (0.0s @ 0.18GB)
Error: ImageSingletons do not contain key com.oracle.svm.core.graal.code.SubstrateLoweringProviderFactory
com.oracle.svm.core.util.UserError$UserException: ImageSingletons do not contain key com.oracle.svm.core.graal.code.SubstrateLoweringProviderFactory
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:73)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ImageSingletonsSupportImpl$HostedManagement.doLookup(ImageSingletonsSupportImpl.java:117)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.ImageSingletonsSupportImpl.lookup(ImageSingletonsSupportImpl.java:44)
        at org.graalvm.sdk/org.graalvm.nativeimage.ImageSingletons.lookup(ImageSingletons.java:86)
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.graal.GraalConfiguration.createLoweringProvider(GraalConfiguration.java:109)
        at org.graalvm.nativeimage.builder/com.oracle.svm.core.graal.meta.SubstrateLoweringProvider.createForHosted(SubstrateLoweringProvider.java:58)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.createBigBang(NativeImageGenerator.java:1090)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:882)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:576)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:536)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:403)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:580)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)

FYI, there are profiles inside the pom.xml which allow building without the LLVM backend override, and with a snapshot of the maven plugin.

Describe GraalVM and your environment:

More details

> ... /22.3.0-dev-20220915_2039/graalvm-ce-java17-22.3.0-dev/bin/gu list

ComponentId              Version             Component name                Stability                     Origin 
---------------------------------------------------------------------------------------------------------------------------------
graalvm                  22.3.0-dev          GraalVM Core                  Supported                     
js                       22.3.0-dev          Graal.js                      Supported                     github.com
llvm                     22.3.0-dev          LLVM Runtime Core             Experimental                  github.com
llvm-toolchain           22.3.0-dev          LLVM.org toolchain            Supported                     github.com
native-image             22.3.0-dev          Native Image                  Early adopter                 github.com
nodejs                   22.3.0-dev          Graal.nodejs                  Supported                     github.com
wasm                     22.3.0-dev          GraalWasm                     Experimental                  github.com
shoffmeister commented 2 years ago

@fniephaus This is a follow up to the Slack conversation https://graalvm.slack.com/archives/CN9KSFB40/p1663426800036569?thread_ts=1663413641.176299&cid=CN9KSFB40 - here focusing on the most recent CE dev build off github.

The 22.2-EE download throws a different error

[1/7] Initializing...                                                                                    (0.0s @ 0.31GB)
Fatal error: java.lang.IllegalArgumentException: package com.oracle.svm.core.graal.llvm not in contents
        at java.base/java.lang.Module.implAddExportsOrOpens(Module.java:940)
        at java.base/java.lang.Module.implAddExports(Module.java:859)
        at java.base/java.lang.System$2.addExports(System.java:2344)
        at java.base/jdk.internal.module.Modules.addExports(Modules.java:105)
        at org.graalvm.nativeimage.base/com.oracle.svm.util.ModuleSupport$Access$2.giveAccess(ModuleSupport.java:74)
        at org.graalvm.nativeimage.base/com.oracle.svm.util.ModuleSupport.accessModuleByClass(ModuleSupport.java:135)
        at org.graalvm.nativeimage.base/com.oracle.svm.util.ModuleSupport.accessPackagesToClass(ModuleSupport.java:122)
        at com.oracle.svm.svm_enterprise.llvm/com.oracle.svm.enterprise.llvm.EnterpriseLLVMFeature.afterRegistration(stripped:36)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.lambda$setupNativeImage$14(NativeImageGenerator.java:840)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FeatureHandler.forEachFeature(FeatureHandler.java:78)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:840)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:561)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:521)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:407)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:585)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)

which I am merely posting for the sake of search engine indexing.

fniephaus commented 2 years ago

As a workaround, you need to install the LLVM backend (via $JAVA_HOME/bin/gu install native-image-llvm-backend). We'll add an actionable error message for this soon.

fniephaus commented 2 years ago

Fix with https://github.com/oracle/graal/commit/1d794eab24b4317bd9d6038b2457efbbd88ad111.

shoffmeister commented 2 years ago

@fniephaus - while this patch obviously fixes the UX for the error message, I don't see how it will address that neither 22-2 (EE) nor 22.3.0-dev-20220915_2039 (CE) are aware of the component native-image-llvm-backend:

22.3-dev (CE):

graalvm/22.3.0-dev-20220915_2039/graalvm-ce-java17-22.3.0-dev on ☁️   
❯ gu available
Downloading: Component catalog from raw.githubusercontent.com
ComponentId              Version             Component name                Stability                     Origin 
---------------------------------------------------------------------------------------------------------------------------------
espresso                 22.3.0-dev          Java on Truffle               Supported                     github.com
espresso-llvm            22.3.0-dev          Java on Truffle LLVM Java librSupported                     github.com
js                       22.3.0-dev          Graal.js                      Supported                     github.com
llvm                     22.3.0-dev          LLVM Runtime Core             Experimental                  github.com
llvm-toolchain           22.3.0-dev          LLVM.org toolchain            Supported                     github.com
native-image             22.3.0-dev          Native Image                  Early adopter                 github.com
nodejs                   22.3.0-dev          Graal.nodejs                  Supported                     github.com
python                   22.3.0-dev          GraalVM Python                Experimental                  github.com
R                        22.3.0-dev          FastR                         Experimental                  github.com
ruby                     22.3.0-dev          TruffleRuby                   Experimental                  github.com
visualvm                 22.3.0-dev          VisualVM                      Experimental                  github.com
wasm                     22.3.0-dev          GraalWasm                     Experimental                  github.com

graalvm/22.3.0-dev-20220915_2039/graalvm-ce-java17-22.3.0-dev on ☁️   
❯ gu install native-image-llvm-backend
Downloading: Component catalog from raw.githubusercontent.com
Error: Component "native-image-llvm-backend" is not available in catalog.

22-2 EE:

graalvm/22-2/graalvm-ee-java17-22.2.0 on ☁️   
❯ gu available
Downloading: Artifacts catalog from gds.oracle.com
Downloading: Release index file from oca.opensource.oracle.com
Downloading: Component catalog for GraalVM Enterprise Edition 22.2.0.1 on jdk17 from oca.opensource.oracle.com
Downloading: Component catalog for GraalVM Enterprise Edition 22.2.0 on jdk17 from oca.opensource.oracle.com
Skipping ULN EE channels, no username provided.
Downloading: Component catalog from www.graalvm.org
ComponentId              Version             Component name                Stability                     Origin 
---------------------------------------------------------------------------------------------------------------------------------
espresso                 22.2.0              Java on Truffle               Supported                     gds.oracle.com
espresso-llvm            22.2.0              Java on Truffle LLVM Java librSupported                     gds.oracle.com
js                       22.2.0              Graal.js                      Supported                     gds.oracle.com
llvm                     22.2.0              LLVM Runtime Core             Experimental                  gds.oracle.com
llvm-toolchain           22.2.0              LLVM.org toolchain            Supported                     gds.oracle.com
native-image             22.2.0              Native Image                  Early adopter                 gds.oracle.com
nodejs                   22.2.0              Graal.nodejs                  Supported                     gds.oracle.com
python                   22.2.0              Graal.Python                  Experimental                  gds.oracle.com
R                        22.2.0              FastR                         Experimental                  github.com
ruby                     22.2.0              TruffleRuby                   Experimental                  gds.oracle.com
visualvm                 22.2.0              VisualVM EE                   Experimental                  gds.oracle.com
wasm                     22.2.0              GraalWasm                     Experimental                  gds.oracle.com

graalvm/22-2/graalvm-ee-java17-22.2.0 on ☁️   
❯ gu install native-image-llvm-backend
Downloading: Artifacts catalog from gds.oracle.com
Downloading: Release index file from oca.opensource.oracle.com
Downloading: Component catalog for GraalVM Enterprise Edition 22.2.0.1 on jdk17 from oca.opensource.oracle.com
Downloading: Component catalog for GraalVM Enterprise Edition 22.2.0 on jdk17 from oca.opensource.oracle.com
Skipping ULN EE channels, no username provided.
Downloading: Component catalog from www.graalvm.org
Error: Component "native-image-llvm-backend" is not available in catalog.
shoffmeister commented 2 years ago

FWIW, https://github.com/oracle/graal/blame/08a2755f8113e7238f45e790709a3a72323a3871/substratevm/mx.substratevm/mx_substratevm.py#L1056 mentions native-image-llvm-backend, but I don't know whether this is enough to get this deployed?

fniephaus commented 2 years ago

I think GraalVM EE 22.2 shouldn't show the warning without further ado, but you do need to install the llvm-toolchain component to use the LLVM backend. With 22.3, EE and CE will both have the native-image-llvm-backend, which will also pull the llvm-toolchain as a dependency.

It seem that the dev build you used is outdated. Can you please retry with the latest dev build, which should have native-image-llvm-backend (e.g., this)?

shoffmeister commented 2 years ago

I just went to https://github.com/graalvm/graalvm-ce-dev-builds/releases and downloaded "GraalVM CE 22.3.0-dev-20221004_1644". This release indeed has gu available listing

native-image-llvm-backend22.3.0-dev          Native Image LLVM Backend     Early adopter (experimental)  github.com

FWIW, I hadn't realized that, in addition to amending the error message, some more action took place to make the machinery available, too.

I'll try to get back at actually exercising this ASAP - many thanks!

fniephaus commented 2 years ago

Thanks for checking, @shoffmeister! Glad this is working now!

asokolov-a1 commented 1 year ago

I ran into the same issue with GraalVM 22.3 with the provided example project.

$ java -version
openjdk version "17.0.5" 2022-10-18
OpenJDK Runtime Environment GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08)
OpenJDK 64-Bit Server VM GraalVM CE 22.3.0 (build 17.0.5+8-jvmci-22.3-b08, mixed mode, sharing)
$ gu list
ComponentId              Version             Component name                Stability                     Origin 
---------------------------------------------------------------------------------------------------------------------------------
graalvm                  22.3.0              GraalVM Core                  Supported                     
llvm-toolchain           22.3.0              LLVM.org toolchain            Supported                     github.com
native-image             22.3.0              Native Image                  Early adopter                 github.com
native-image-llvm-backend22.3.0              Native Image LLVM Backend     Early adopter (experimental)  github.com
syrusakbary commented 1 year ago

native-image-llvm-backend doesn't seem to be available in macOS Aarch64 (ARM/M1). Any plans for having it there?

loicottet commented 1 year ago

We are in the process of supporting the LLVM backend on ARM macOS, these builds should be available soon