search

apache / incubator-kie-issues

Apache License 2.0
12 stars 1 forks source link

kogito-examples: pmml-quarkus-example and other pmml examples fail to build native image #1441

Closed tkobayas closed 2 months ago

tkobayas commented 3 months ago 
cd kogito-quarkus-examples/pmml-quarkus-example
mvn clean package -Dnative -Dquarkus.native.container-build=true
========================================================================================================================
GraalVM Native Image: Generating 'pmml-quarkus-example-runner' (executable)...
========================================================================================================================
For detailed information and explanations on the build output, visit:
https://github.com/oracle/graal/blob/master/docs/reference-manual/native-image/BuildOutput.md
------------------------------------------------------------------------------------------------------------------------
[1/8] Initializing...                                                                                    (8.9s @ 0.23GB)
 Java version: 21.0.4+7-LTS, vendor version: Mandrel-23.1.4.0-Final
 Graal compiler: optimization level: 2, target machine: x86-64-v3
 C compiler: gcc (redhat, x86_64, 8.5.0)
 Garbage collector: Serial GC (max heap size: 80% of RAM)
 4 user-specific feature(s):
 - com.oracle.svm.thirdparty.gson.GsonFeature
 - io.quarkus.runner.Feature: Auto-generated class by Quarkus from the existing extensions
 - io.quarkus.runtime.graal.DisableLoggingFeature: Disables INFO logging during the analysis phase
 - org.eclipse.angus.activation.nativeimage.AngusActivationFeature
------------------------------------------------------------------------------------------------------------------------
 3 experimental option(s) unlocked:
 - '-H:+AllowFoldMethods' (origin(s): command line)
 - '-H:BuildOutputJSONFile' (origin(s): command line)
 - '-H:+UseServiceLoaderFeature' (origin(s): command line)
------------------------------------------------------------------------------------------------------------------------
Build resources:
 - 7.11GB of memory (22.9% of 31.10GB system memory, set via '-Xmx8g')
 - 8 thread(s) (100.0% of 8 available processor(s), determined at start)
[2/8] Performing analysis...  [******]                                                                  (85.9s @ 2.67GB)
   26,397 reachable types   (89.4% of   29,524 total)
   38,942 reachable fields  (61.0% of   63,853 total)
  132,794 reachable methods (62.4% of  212,717 total)
    7,621 types,   544 fields, and 6,730 methods registered for reflection
       61 types,    61 fields, and    55 methods registered for JNI access
        4 native libraries: dl, pthread, rt, z

Error: Type is not available in this platform: com.oracle.svm.hosted.NativeImageClassLoader$1
Detailed message:
Trace: Object was reached by
  reading field java.util.ServiceLoader$LazyClassPathLookupIterator.configs of constant 
    java.util.ServiceLoader$LazyClassPathLookupIterator@7a9e8687: java.util.ServiceLoader$LazyClassPathLookupIterator@7a9e8687
  reading field java.util.ServiceLoader$2.val$second of constant 
    java.util.ServiceLoader$2@157aafa5: java.util.ServiceLoader$2@157aafa5
  reading field java.util.ServiceLoader.lookupIterator1 of constant 
    java.util.ServiceLoader@7e6f18ec: java.util.ServiceLoader[org.kie.efesto.runtimemanager.api.service.RuntimeManager...
  scanning root java.util.ServiceLoader@7e6f18ec: java.util.ServiceLoader[org.kie.efesto.runtimemanager.api.service.RuntimeManager... embedded in 
    org.kie.efesto.runtimemanager.api.utils.SPIUtils.getManagers(SPIUtils.java:108)
  parsing method org.kie.efesto.runtimemanager.api.utils.SPIUtils.getManagers(SPIUtils.java:107) reachable via the parsing context
    at org.kie.efesto.runtimemanager.api.utils.SPIUtils.getRuntimeManager(SPIUtils.java:94)
    at org.kie.pmml.models.drools.commons.model.KiePMMLDroolsModel.evaluate(KiePMMLDroolsModel.java:117)
    at org.kie.pmml.models.drools.executor.DroolsModelEvaluator.evaluate(DroolsModelEvaluator.java:40)
    at org.kie.pmml.models.drools.executor.DroolsModelEvaluator.evaluate(DroolsModelEvaluator.java:32)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.evaluate(PMMLRuntimeHelper.java:171)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.evaluate(PMMLRuntimeHelper.java:225)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.getEfestoOutput(PMMLRuntimeHelper.java:180)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.executeEfestoInputPMML(PMMLRuntimeHelper.java:104)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.executeEfestoInput(PMMLRuntimeHelper.java:124)
    at org.kie.pmml.evaluator.core.service.KieRuntimeServicePMMLRequestData.evaluateInput(KieRuntimeServicePMMLRequestData.java:52)
    at root method.(Unknown Source)

com.oracle.svm.core.util.UserError$UserException: Type is not available in this platform: com.oracle.svm.hosted.NativeImageClassLoader$1
Detailed message:
Trace: Object was reached by
  reading field java.util.ServiceLoader$LazyClassPathLookupIterator.configs of constant 
    java.util.ServiceLoader$LazyClassPathLookupIterator@7a9e8687: java.util.ServiceLoader$LazyClassPathLookupIterator@7a9e8687
  reading field java.util.ServiceLoader$2.val$second of constant 
    java.util.ServiceLoader$2@157aafa5: java.util.ServiceLoader$2@157aafa5
  reading field java.util.ServiceLoader.lookupIterator1 of constant 
    java.util.ServiceLoader@7e6f18ec: java.util.ServiceLoader[org.kie.efesto.runtimemanager.api.service.RuntimeManager...
  scanning root java.util.ServiceLoader@7e6f18ec: java.util.ServiceLoader[org.kie.efesto.runtimemanager.api.service.RuntimeManager... embedded in 
    org.kie.efesto.runtimemanager.api.utils.SPIUtils.getManagers(SPIUtils.java:108)
  parsing method org.kie.efesto.runtimemanager.api.utils.SPIUtils.getManagers(SPIUtils.java:107) reachable via the parsing context
    at org.kie.efesto.runtimemanager.api.utils.SPIUtils.getRuntimeManager(SPIUtils.java:94)
    at org.kie.pmml.models.drools.commons.model.KiePMMLDroolsModel.evaluate(KiePMMLDroolsModel.java:117)
    at org.kie.pmml.models.drools.executor.DroolsModelEvaluator.evaluate(DroolsModelEvaluator.java:40)
    at org.kie.pmml.models.drools.executor.DroolsModelEvaluator.evaluate(DroolsModelEvaluator.java:32)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.evaluate(PMMLRuntimeHelper.java:171)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.evaluate(PMMLRuntimeHelper.java:225)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.getEfestoOutput(PMMLRuntimeHelper.java:180)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.executeEfestoInputPMML(PMMLRuntimeHelper.java:104)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.executeEfestoInput(PMMLRuntimeHelper.java:124)
    at org.kie.pmml.evaluator.core.service.KieRuntimeServicePMMLRequestData.evaluateInput(KieRuntimeServicePMMLRequestData.java:52)
    at root method.(Unknown Source)

    at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:85)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.FallbackFeature.reportAsFallback(FallbackFeature.java:248)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:814)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:592)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:550)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:539)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:721)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.start(NativeImageGeneratorRunner.java:143)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:98)
Caused by: com.oracle.graal.pointsto.constraints.UnsupportedFeatureException: Type is not available in this platform: com.oracle.svm.hosted.NativeImageClassLoader$1
Detailed message:
Trace: Object was reached by
  reading field java.util.ServiceLoader$LazyClassPathLookupIterator.configs of constant 
    java.util.ServiceLoader$LazyClassPathLookupIterator@7a9e8687: java.util.ServiceLoader$LazyClassPathLookupIterator@7a9e8687
  reading field java.util.ServiceLoader$2.val$second of constant 
    java.util.ServiceLoader$2@157aafa5: java.util.ServiceLoader$2@157aafa5
  reading field java.util.ServiceLoader.lookupIterator1 of constant 
    java.util.ServiceLoader@7e6f18ec: java.util.ServiceLoader[org.kie.efesto.runtimemanager.api.service.RuntimeManager...
  scanning root java.util.ServiceLoader@7e6f18ec: java.util.ServiceLoader[org.kie.efesto.runtimemanager.api.service.RuntimeManager... embedded in 
    org.kie.efesto.runtimemanager.api.utils.SPIUtils.getManagers(SPIUtils.java:108)
  parsing method org.kie.efesto.runtimemanager.api.utils.SPIUtils.getManagers(SPIUtils.java:107) reachable via the parsing context
    at org.kie.efesto.runtimemanager.api.utils.SPIUtils.getRuntimeManager(SPIUtils.java:94)
    at org.kie.pmml.models.drools.commons.model.KiePMMLDroolsModel.evaluate(KiePMMLDroolsModel.java:117)
    at org.kie.pmml.models.drools.executor.DroolsModelEvaluator.evaluate(DroolsModelEvaluator.java:40)
    at org.kie.pmml.models.drools.executor.DroolsModelEvaluator.evaluate(DroolsModelEvaluator.java:32)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.evaluate(PMMLRuntimeHelper.java:171)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.evaluate(PMMLRuntimeHelper.java:225)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.getEfestoOutput(PMMLRuntimeHelper.java:180)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.executeEfestoInputPMML(PMMLRuntimeHelper.java:104)
    at org.kie.pmml.evaluator.core.utils.PMMLRuntimeHelper.executeEfestoInput(PMMLRuntimeHelper.java:124)
    at org.kie.pmml.evaluator.core.service.KieRuntimeServicePMMLRequestData.evaluateInput(KieRuntimeServicePMMLRequestData.java:52)
    at root method.(Unknown Source)

    at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.constraints.UnsupportedFeatures.report(UnsupportedFeatures.java:126)
    at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.runPointsToAnalysis(NativeImageGenerator.java:809)
    ... 6 more

You can confirm it locally and also in nightly.native jobs

https://ci-builds.apache.org/job/KIE/job/kogito/job/main/job/nightly.native/job/kogito-examples.build-and-test/ https://ci-builds.apache.org/job/KIE/job/kogito/job/10.0.x/job/nightly.native/job/kogito-examples.build-and-test/

tkobayas commented 3 months ago

Another error variant:

[2024-08-16T02:31:36.570Z] Error: The heap snapshot verifier discovered a type not marked as reachable: AnalysisType<Aggregate$Function[] -> HotSpotType<[Lorg/dmg/pmml/Aggregate$Function;, resolved>, allocated: false, inHeap: false, reachable: false>
[2024-08-16T02:31:36.570Z] com.oracle.graal.pointsto.util.AnalysisError: The heap snapshot verifier discovered a type not marked as reachable: AnalysisType<Aggregate$Function[] -> HotSpotType<[Lorg/dmg/pmml/Aggregate$Function;, resolved>, allocated: false, inHeap: false, reachable: false>
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.guarantee(AnalysisError.java:184)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.heap.HeapSnapshotVerifier$ScanningObserver.ensureTypeScanned(HeapSnapshotVerifier.java:332)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.heap.HeapSnapshotVerifier$ScanningObserver.forScannedConstant(HeapSnapshotVerifier.java:318)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.scanConstant(ObjectScanner.java:252)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.scanField(ObjectScanner.java:181)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.doScan(ObjectScanner.java:386)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.ObjectScanner.lambda$scanConstant$6(ObjectScanner.java:257)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
[2024-08-16T02:31:36.570Z]      at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
[2024-08-16T02:31:36.570Z]      at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
[2024-08-16T02:31:36.570Z]      at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
[2024-08-16T02:31:36.570Z]      at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
[2024-08-16T02:31:36.570Z]      at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
[2024-08-16T02:31:36.570Z]      at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
[2024-08-16T02:31:36.570Z]      at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
tkobayas commented 2 months ago

Note: stable branch (kogito 1.44.1.Final, drools 8.44.0.Final) can successfully build native image.

tkobayas commented 2 months ago 
mvn clean package -Dnative -Dquarkus.native.container-build=true -Dquarkus.native.builder-image=quay.io/quarkus/ubi-quarkus-mandrel-builder-image:22.3-java17

The old builder image quay.io/quarkus/ubi-quarkus-mandrel-builder-image:22.3-java17 can successfully build native image.

So the issue seems to be introduced by the new builder image quay.io/quarkus/ubi-quarkus-mandrel-builder-image:jdk-21

tkobayas commented 2 months ago

I'll keep investigating.

@gitgabrio @mariofusco , if you have any thoughts, please share, thanks!

gitgabrio commented 2 months ago

@tkobayas looking...

gitgabrio commented 2 months ago

@tkobayas @mariofusco @baldimir Surely the new Graal version changed somehow its discovery mechanism. I could recreate locally using the Graal 21 version, and I verified the problems on pmml-quarks-example and dmn-pmml-quarkus-example.

The problem was due to the presence of old pmml-drools dependencies. I created two PRs to fix it (I tested locally, and it worked, but let's wait CI to be 100% sure) kogito-runtimes kogito-examples

The one in kogito-example should be temporary, because the root problem is the existence of pmml-drools artifacts (that should be completely deleted, but I had no time/commitment to do).

porcelli commented 2 months ago

@tkobayas I wonder why we have GraalVM 21 here instead 17?

LightGuard commented 2 months ago

https://quarkus.io/version/3.8/guides/building-native-image#configuring-graalvm I don't know why the Quarkus website says Graal 21, but Java 17+

tkobayas commented 2 months ago

@porcelli @LightGuard

https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/Java.2017.20native.20build.20compatibility.20expectation.20for.203.2E8.2Ex.2B.3F

we still target java 17 for jvm mode

and same for native image - just the runtime is 21 based.

...

IIRC, we decided to only support the 21 builder for GraalVM/Mandrel

So it seems to be intentional. (just wanted to keep up with the latest runtime?)