quarkusio / quarkus

Quarkus: Supersonic Subatomic Java.
https://quarkus.io
Apache License 2.0
13.64k stars 2.65k forks source link

Mongo top-level class with generics broken #36397

Open GregJohnStewart opened 12 months ago

GregJohnStewart commented 12 months ago

Describe the bug

The more recent versions of Quarkus have seemed to break how the PojoCodec handles classes with generics (even only when being passed instantiated classes with fully realized generics). This was supported and works fine in 3.2.4.Final.

However, when moving to 3.4.2, the following error occurs:

org.bson.codecs.configuration.CodecConfigurationException: Animal contains generic types that have not been specialised. Top level classes with generic types are not supported by the PojoCodec.

As I assumed first that this was a codec issue, I have a ticket out on Mongo's jira, but they claim not a ton has changed: https://jira.mongodb.org/browse/JAVA-5173

This prevents me moving to newer Quarkus versions, and also breaks a super handy feature of Mongodb being able to easily handle polymorphism

Expected behavior

the POJO Codec should be able a class with generics, with a @BsonDiscriminator to manage the concrete class handling

Actual behavior

The POJO Codec fails with the following error:

org.bson.codecs.configuration.CodecConfigurationException: Animal contains generic types that have not been specialised.
Top level classes with generic types are not supported by the PojoCodec.
    at org.bson.codecs.pojo.LazyPropertyModelCodec$NeedSpecializationCodec.exception(LazyPropertyModelCodec.java:180)
    at org.bson.codecs.pojo.LazyPropertyModelCodec$NeedSpecializationCodec.decode(LazyPropertyModelCodec.java:166)
    at com.mongodb.internal.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52)
    at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:60)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42)
    at org.bson.internal.LazyCodec.decode(LazyCodec.java:53)
    at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:104)
    at com.mongodb.internal.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:63)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:87)
    at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:42)
    at com.mongodb.internal.connection.ReplyMessage.<init>(ReplyMessage.java:48)
    at com.mongodb.internal.connection.InternalStreamConnection.getCommandResult(InternalStreamConnection.java:565)
    at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:455)
    at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:370)
    at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:114)
    at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:719)
    at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:76)
    at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:203)
    at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:115)
    at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:83)
    at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:74)
    at com.mongodb.internal.connection.DefaultServer$OperationCountTrackingConnection.command(DefaultServer.java:287)
    at com.mongodb.internal.operation.CommandOperationHelper.createReadCommandAndExecute(CommandOperationHelper.java:245)
    at com.mongodb.internal.operation.FindOperation.lambda$execute$1(FindOperation.java:324)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$0(OperationHelper.java:345)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370)
    at com.mongodb.internal.operation.OperationHelper.lambda$withSourceAndConnection$1(OperationHelper.java:344)
    at com.mongodb.internal.operation.OperationHelper.withSuppliedResource(OperationHelper.java:370)
    at com.mongodb.internal.operation.OperationHelper.withSourceAndConnection(OperationHelper.java:343)
    at com.mongodb.internal.operation.FindOperation.lambda$execute$2(FindOperation.java:321)
    at com.mongodb.internal.operation.CommandOperationHelper.lambda$decorateReadWithRetries$3(CommandOperationHelper.java:192)
    at com.mongodb.internal.async.function.RetryingSyncSupplier.get(RetryingSyncSupplier.java:67)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:332)
    at com.mongodb.internal.operation.FindOperation.execute(FindOperation.java:72)
    at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:153)
    at com.mongodb.client.internal.FindIterableImpl.first(FindIterableImpl.java:213)
    at org.acme.AnimalServiceTest.testAnimalService(AnimalServiceTest.java:28)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:577)
    at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:1015)
    at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:829)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
    at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
    at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
    at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:110)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:90)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:85)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:62)
    at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
    at java.base/java.lang.reflect.Method.invoke(Method.java:577)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at jdk.proxy1/jdk.proxy1.$Proxy2.stop(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
    at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

How to Reproduce?

https://github.com/GregJohnStewart/quarkus-proofs/tree/main/mongo-codec-issue

./gradlew test

Switch to see the tests working under 3.2.4.Final in gradle.properties

Output of uname -a or ver

No response

Output of java -version

openjdk 17.0.8.1 2023-08-24 OpenJDK Runtime Environment (build 17.0.8.1+1-Ubuntu-0ubuntu120.04) OpenJDK 64-Bit Server VM (build 17.0.8.1+1-Ubuntu-0ubuntu120.04, mixed mode, sharing)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

3.2.4.Final

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle

Additional information

On a side note, I have noticed a few other issues with the codec (even in the older versions), from not properly deserializing (returning classes of the wrong type)(haven't seen this consistently reproducibly enough to report), as well as the codec not being able to reconcile and deserialize classes after a restart in dev mode (reproducible, more minor, if highly annoying). Can provide additional details, or another issue if needed. Just throwing it out there the latest iterations of the mongo client seem off.

quarkus-bot[bot] commented 12 months ago

/cc @evanchooly (mongodb), @loicmathieu (mongodb)

loicmathieu commented 11 months ago

Hi, Reading the ticket at MongoDB side you dissect the issue being the update of the MongoDB driver from 4.9.1 to 4.10.1. So the issue seems to be on the MongoDB driver and not on the Quarkus MongoDB client extension. You can confirm that by forcing on your test application an older version of the driver.

GregJohnStewart commented 11 months ago

That was my assumption, that it was the updated driver. However, it looks like (from that mongodb ticket) that not much has changed to the pojo codec.

My assumption at this point (hence this ticket) is not that the codec is at fault, but how it is setup within Quarkus. Setting the quarkus version back to 3.2.4.Final, and running the tests show that they passed on earlier quarkus versions. Thanks for the check on the codec version though!

(If you are using the same POC code to test again, do a pull, I made some tweaks to pass once it got past the error reported here)

loicmathieu commented 11 months ago

What changed between 3.2 and 3.3 is the version of the MongoDB driver (from 4.9 to 4.10), as far as I remember (but I didn't check) nothing more changed.

That's why I ask if you can try with Quarkus 3.3 or 3.4 by downgrading the MongoDB driver to 4.9 as I think the issue is in the driver. If it proves to not be the case I'll be happy to dig deeper.

GregJohnStewart commented 11 months ago

Ah apologies, trying now. Happen to know how to do this without having to rebuild the Quarkus plugin? Even when looking at the plugin though, where are library versions defined? There are no versions listed for dependencies under extensions/mongodb-client/runtime. Poking around, looks like versions are centrally managed somewhere, but not seeing where.

loicmathieu commented 11 months ago

You must use dependency management to force a different version of MongoDB of the one coming from the Quarkus BOM.

I'm not familiar with Gradle but there is some strategy explained here: https://docs.gradle.org/current/userguide/dependency_downgrade_and_exclude.html

GregJohnStewart commented 11 months ago

I made several attempts, but no joy on getting past a gradle verification step on forcing the lib version.. (various libs that use the core versions were disliking the change)

I just pushed the addition of a pom and wrapper so should be good to go as a Maven project. I don't know offhand how to force the versioning here though. I can do some googling, but if @loicmathieu, you know offhand the dive would be appreciated

loicmathieu commented 11 months ago

On the pom.xml you add a the dependency inside <dependencyManagement> with the forced version. If I remember correctly this should be before the Quarkus BOM so it is taken into consideration.

GregJohnStewart commented 11 months ago

Alright, after much tweaking, I think I finally have the Mongo dependency forced to 4.9.1 (on Gradle)(I'm willing to be proven that I did this wrong, but I think I am right with it). The changes have been pushed.

Alas, the issue still occurs with the driver version forced back to 4.9.1, and using Quarkus 3.5.0.

Sorry for the lateness, @loicmathieu life has been busy

loicmathieu commented 11 months ago

@GregJohnStewart thanks for making the verification, we need to investigate what's going on.

loicmathieu commented 11 months ago

I can reproduce it.

For the record, in Gradle, the easiest way to force a dependency version is the following:

configurations.all {
    resolutionStrategy {
        force('org.mongodb:mongodb-driver-sync:4.9.1')
        force('org.mongodb:mongodb-driver-reactivestreams:4.9.1')
        force('org.mongodb:mongodb-driver-core:4.9.1')
    }
}
GregJohnStewart commented 8 months ago

@loicmathieu :wave: Happen to dive into this? Been a hot second. Still an issue as of 3.6.6.

GregJohnStewart commented 6 months ago

Still an issue moving to 3.9.2

ashni-mongodb commented 4 days ago

Hi folks, we've included a fix for this regression in a recent release of the MongoDB Java Drivers (PR: https://github.com/mongodb/mongo-java-driver/pull/1423). Please have a look, and let me know if you have feedback or thoughts. Thank you!

loicmathieu commented 1 day ago

@ashni-mongodb thanks for the information, I'll check our reproducer with the latest MongoDB driver and give feedback