search

quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.38k stars 2.56k forks source link

Quarkus 3.12.1 + Kafka + DataDog + Native = BOOM #41699

Open luneo7 opened 6 days ago

luneo7 commented 6 days ago

Describe the bug

Using Quarku 3.12.1 + Kafka + DD when building native we randomly get errors as:

Fatal error: com.oracle.graal.pointsto.util.AnalysisError$ParsingError: Error encountered while parsing org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.$get$__datadogContext$(KafkaAdminClient.java) 
Parsing context:
   at datadog.trace.bootstrap.FieldBackedContextStore.get(FieldBackedContextStore.java:17)
   at datadog.trace.bootstrap.instrumentation.java.concurrent.AdviceUtils.capture(AdviceUtils.java:65)
   at java.util.concurrent.ForkJoinTask.fork(ForkJoinTask.java:626)
   at java.math.BigInteger$RecursiveOp.forkOrInvoke(BigInteger.java:1918)
   at java.math.BigInteger$RecursiveOp.square(BigInteger.java:1960)
   at java.math.BigInteger.squareToomCook3(BigInteger.java:2383)
   at java.math.BigInteger.square(BigInteger.java:2227)
   at java.math.BigInteger.multiply(BigInteger.java:1637)
   at java.math.BigInteger.multiply(BigInteger.java:1594)
   at java.math.MutableBigInteger.divide3n2n(MutableBigInteger.java:1370)
   at java.math.MutableBigInteger.divide2n1n(MutableBigInteger.java:1335)
   at java.math.MutableBigInteger.divideAndRemainderBurnikelZiegler(MutableBigInteger.java:1290)
   at java.math.MutableBigInteger.divide(MutableBigInteger.java:1160)
   at java.math.BigInteger.smallToString(BigInteger.java:4164)
   at java.math.BigInteger.toString(BigInteger.java:4217)
   at java.math.BigInteger.toString(BigInteger.java:4112)
   at java.math.BigInteger.toString(BigInteger.java:4286)
   at java.util.Formatter$FormatSpecifier.print(Formatter.java:3451)
   at java.util.Formatter$FormatSpecifier.printInteger(Formatter.java:3064)
   at java.util.Formatter$FormatSpecifier.print(Formatter.java:3021)
   at java.util.Formatter.format(Formatter.java:2780)
   at java.util.Formatter.format(Formatter.java:2728)
   at java.lang.String.format(String.java:4390)
   at jdk.internal.util.Preconditions.outOfBoundsMessage(Preconditions.java:242)
   at jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)
   at jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)
   at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)
   at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:106)
   at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:302)
   at java.lang.StringUTF16.checkIndex(StringUTF16.java:1613)
   at java.lang.StringUTF16.charAt(StringUTF16.java:1418)
   at java.lang.String.charAt(String.java:1557)
   at java.lang.Long.parseLong(Long.java:689)
   at java.lang.Long.valueOf(Long.java:1157)
   at static root method.(Unknown Source)

        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.parsingError(AnalysisError.java:149)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:184)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:153)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraphInfo(MethodTypeFlow.java:111)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultVirtualInvokeTypeFlow.java:114)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.TypeFlow.update(TypeFlow.java:620)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.PointsToAnalysis$1.run(PointsToAnalysis.java:491)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.executeCommand(CompletionExecutor.java:187)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.CompletionExecutor.lambda$executeService$0(CompletionExecutor.java:171)
        at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1312)
        at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1843)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1808)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:188)
Caused by: com.oracle.graal.pointsto.util.AnalysisError: parsing had failed in another thread
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.util.AnalysisError.shouldNotReachHere(AnalysisError.java:173)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsedHelper(AnalysisMethod.java:871)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsed(AnalysisMethod.java:843)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:186)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:621)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:167)
        ... 13 more
Caused by: com.oracle.graal.pointsto.constraints.UnresolvedElementException: Discovered unresolved field during parsing: org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.__datadogContext$10. This error is reported at image build time because class org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable is registered for linking at image build time by command line and command line.
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.reportUnresolvedElement(SharedGraphBuilderPhase.java:550)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.reportUnresolvedElement(SharedGraphBuilderPhase.java:544)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.handleUnresolvedField(SharedGraphBuilderPhase.java:453)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.handleUnresolvedLoadField(SharedGraphBuilderPhase.java:363)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genGetField(BytecodeParser.java:4776)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.genGetField(BytecodeParser.java:4767)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBytecode(BytecodeParser.java:5412)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.iterateBytecodesForBlock(BytecodeParser.java:3426)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.iterateBytecodesForBlock(SharedGraphBuilderPhase.java:743)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.handleBytecodeBlock(BytecodeParser.java:3386)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.processBlock(BytecodeParser.java:3228)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.build(BytecodeParser.java:1137)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase$SharedBytecodeParser.build(SharedGraphBuilderPhase.java:162)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.BytecodeParser.buildRootMethod(BytecodeParser.java:1029)
        at jdk.internal.vm.compiler/org.graalvm.compiler.java.GraphBuilderPhase$Instance.run(GraphBuilderPhase.java:101)
        at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.phases.SharedGraphBuilderPhase.run(SharedGraphBuilderPhase.java:116)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.run(Phase.java:49)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.BasePhase.apply(BasePhase.java:434)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:42)
        at jdk.internal.vm.compiler/org.graalvm.compiler.phases.Phase.apply(Phase.java:38)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.AnalysisParsedGraph.parseBytecode(AnalysisParsedGraph.java:146)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.parseGraph(AnalysisMethod.java:895)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsedHelper(AnalysisMethod.java:860)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.meta.AnalysisMethod.ensureGraphParsed(AnalysisMethod.java:843)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.phases.InlineBeforeAnalysisGraphDecoder.lookupEncodedGraph(InlineBeforeAnalysisGraphDecoder.java:198)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.doInline(PEGraphDecoder.java:1211)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.tryInline(PEGraphDecoder.java:1194)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.trySimplifyInvoke(PEGraphDecoder.java:1049)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.handleInvokeWithCallTarget(PEGraphDecoder.java:1001)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.handleInvoke(PEGraphDecoder.java:987)
        at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphDecoder.processNextNode(GraphDecoder.java:921)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.phases.InlineBeforeAnalysisGraphDecoder.processNextNode(InlineBeforeAnalysisGraphDecoder.java:378)
        at jdk.internal.vm.compiler/org.graalvm.compiler.nodes.GraphDecoder.decode(GraphDecoder.java:650)
        at jdk.internal.vm.compiler/org.graalvm.compiler.replacements.PEGraphDecoder.decode(PEGraphDecoder.java:892)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.phases.InlineBeforeAnalysis.decodeGraph(InlineBeforeAnalysis.java:76)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.parse(MethodTypeFlowBuilder.java:195)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlowBuilder.apply(MethodTypeFlowBuilder.java:621)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.createFlowsGraph(MethodTypeFlow.java:167)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.ensureFlowsGraphCreated(MethodTypeFlow.java:153)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.MethodTypeFlow.getOrCreateMethodFlowsGraphInfo(MethodTypeFlow.java:111)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.typestate.DefaultVirtualInvokeTypeFlow.onObservedUpdate(DefaultVirtualInvokeTypeFlow.java:114)
        at org.graalvm.nativeimage.pointsto/com.oracle.graal.pointsto.flow.TypeFlow.lambda$addObserver$0(TypeFlow.java:475)
        ... 8 more

Expected behavior

Build passes as with Quarkus 3.8

Actual behavior

Build fails with random Discovered unresolved field during parsing ... __datadogContext error, with random classes.

How to Reproduce?

  1. Clone https://github.com/luneo7/kafka-with-quarkus
  2. Update application.properties quarkus.native.container-runtime-options config so the path to dd-java-agent-1.35.2.jar is the correct one

  3. Run the build with mvn clean package -Dquarkus.native.container-build=true -Pnative -DskipTests=true

    Sometimes the build passes, so you might need to kick it a few times... funny thing is that in the CI it fails (almost all the time) more than locally...

Output of uname -a or ver

Darwin C02C32WQMD6R 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:09:52 PDT 2024; root:xnu-10063.121.3~5/RELEASE_X86_64 x86_64

Output of java -version

openjdk version "21.0.3" 2024-04-16 LTS OpenJDK Runtime Environment Liberica-NIK-23.1.3-2 (build 21.0.3+10-LTS) OpenJDK 64-Bit Server VM Liberica-NIK-23.1.3-2 (build 21.0.3+10-LTS, mixed mode, sharing)

Mandrel or GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.12.1

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.9.5 (57804ffe001d7215b5e7bcb531cf83df38f93546) Maven home: /usr/local/Cellar/maven/3.9.5/libexec Java version: 21.0.3, vendor: BellSoft, runtime: /Library/Java/JavaVirtualMachines/bellsoft-liberica-vm-full-openjdk21-23.1.3/Contents/Home Default locale: en_CA, platform encoding: UTF-8 OS name: "mac os x", version: "14.5", arch: "x86_64", family: "mac"

Additional information

Also created an issue in DD: https://github.com/DataDog/dd-trace-java/issues/7281

quarkus-bot[bot] commented 6 days ago

/cc @Karm (mandrel), @alesj (kafka), @cescoffier (kafka), @galderz (mandrel), @ozangunalp (kafka), @zakkak (mandrel,native-image)

luneo7 commented 6 days ago

I know that we had some changes related to class loading, but they were reverted in 3.12.1 right? As with Quarkus 3.8 it works as expected, but with 3.12.1 it doesn't work... so dunno if it might be to class loading changes...

luneo7 commented 6 days ago

So, this happens since 3.9 =] Tested, up to 3.8 up to 3.8.5 and there it doesn't happen...

geoand commented 5 days ago

This is highly unlikely to be a Quarkus issue