Deserialization Errors Caused by Avro Classes being in the Wrong ClassLoader for Tests

briancullen commented 11 months ago

Describe the bug

We are previously using quarkus versions less that < 2.8 and are trying to upgrade to the latest version but we are having issues with the changes that have occurred with how AVRO is handled.

We are using gradle with a number of subprojects. As these services defined by these subprojects tend to communicate over kafka we separated out the avro (along with some related Java classes) to it's own subproject which the consumers and producers then add as an implementation dependency. Unfortunately when we try this on newer version of quarkus, specifically those that print an error message requiring us to import the quarkus-confluent-registry-avro extension things break.This seems to be after Quarkus 2.8

It appears that this is just a class loader issue related to tests as the service runs fine otherwise.

Expected behavior

Integration tests should be able to serialize and deserialize avro objects that are also used by non-test code.

Actual behavior

java.lang.ClassCastException: class org.acme.kafka.quarkus.Movie cannot be cast to class org.apache.avro.specific.SpecificRecord (org.acme.kafka.quarkus.Movie is in unnamed module of loader 'app'; org.apache.avro.specific.SpecificRecord is in unnamed module of loader io.quarkus.bootstrap.classloading.QuarkusClassLoader @435871cb)
    at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.getSpecificReaderSchema(AbstractKafkaAvroDeserializer.java:287)
    at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.getReaderSchema(AbstractKafkaAvroDeserializer.java:253)
    at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.getDatumReader(AbstractKafkaAvroDeserializer.java:193)
    at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.read(AbstractKafkaAvroDeserializer.java:392)
    at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:114)
    at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:53)
    at org.apache.kafka.common.serialization.Deserializer.deserialize(Deserializer.java:60)
    at io.smallrye.reactive.messaging.kafka.fault.DeserializerWrapper.lambda$deserialize$1(DeserializerWrapper.java:77)
    at io.smallrye.reactive.messaging.kafka.fault.DeserializerWrapper.wrapDeserialize(DeserializerWrapper.java:109)
    at io.smallrye.reactive.messaging.kafka.fault.DeserializerWrapper.deserialize(DeserializerWrapper.java:77)
    at org.apache.kafka.clients.consumer.internals.CompletedFetch.parseRecord(CompletedFetch.java:303)
    at org.apache.kafka.clients.consumer.internals.CompletedFetch.fetchRecords(CompletedFetch.java:264)
    at org.apache.kafka.clients.consumer.internals.AbstractFetch.fetchRecords(AbstractFetch.java:340)
    at org.apache.kafka.clients.consumer.internals.AbstractFetch.collectFetch(AbstractFetch.java:306)
    at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1330)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1254)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1227)
    at io.smallrye.reactive.messaging.kafka.impl.ReactiveKafkaConsumer.lambda$poll$4(ReactiveKafkaConsumer.java:187)
    at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
    at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:36)
    at io.smallrye.mutiny.operators.uni.UniOperatorProcessor.onItem(UniOperatorProcessor.java:47)
    at io.smallrye.mutiny.operators.uni.UniMemoizeOp.forwardTo(UniMemoizeOp.java:123)
    at io.smallrye.mutiny.operators.uni.UniMemoizeOp.subscribe(UniMemoizeOp.java:67)
    at io.smallrye.mutiny.operators.AbstractUni.subscribe(AbstractUni.java:36)
    at io.smallrye.mutiny.operators.uni.UniRunSubscribeOn.lambda$subscribe$0(UniRunSubscribeOn.java:27)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)

How to Reproduce?

Download the master branch of https://github.com/briancullen/quakus-avro-sample and run the integration tests for either the consumer or the producer.

Output of `uname -a` or `ver`

No response

Output of `java -version`

openjdk version "17.0.2" 2022-01-18

Quarkus version or git rev

3.5.0

Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Gradle 8.2.1

Additional information

This was originally reported in https://github.com/quarkusio/quarkus/issues/31650 but I was unable to make the reproducer work at the time. In the end we didn't need to move the service involved but this has now become an issue with a different service.

briancullen commented 11 months ago

Only just came across this https://github.com/quarkusio/quarkus/issues/16804 which seems to be the same issue. It does appear that setting quarkus.test.flat-class-path=true in the application.properties does sort the issue.

If that is the expected way of doing it we can close this ticket although I don't know if it would be worth updating the documentation in that case?

yrodiere commented 11 months ago

Hey @ppalaga, would you have some insight about this?

ppalaga commented 11 months ago

Hey @ppalaga, would you have some insight about this?

Sorry, #16804 was long time ago. Moreover, it was solved by Stuart who has not posted any explanation back then. Not sure who is the current class loading subject matter expert. @maxandersen might know?

quarkusio / quarkus