search

apache / pinot

Apache Pinot - A realtime distributed OLAP datastore
https://pinot.apache.org/
Apache License 2.0
5.26k stars 1.23k forks source link

NoSuchMethodError: AbstractKafkaAvroDeserializer.executeRules #12785

Open cvinacuasonoc opened 3 months ago

cvinacuasonoc commented 3 months ago

On Pinot version 1.2.0-SNAPSHOT-3697552290f64b62e938152b891764adfb0ee031 We get a NoSuchMethodError when ingesting Avro messages from Kafka. The same configuration works fine on Pinot version 1.0.0

1 2024-04-03T10:26:01.458004170+00:00 stderr F at java.base/java.lang.Thread.run(Thread.java:829) 2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:703) 2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.consumeLoop(RealtimeSegmentDataManager.java:465) 2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.processStreamEvents(RealtimeSegmentDataManager.java:576) 2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.spi.stream.StreamDataDecoderImpl.decode(StreamDataDecoderImpl.java:54) 2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:53) 2024-04-03 12:26:01.611 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:121) 2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:111) 2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:103) 2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:108) 2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:188) 2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.read(AbstractKafkaAvroDeserializer.java:532) 2024-04-03 12:26:01.610 2024-04-03T10:26:01.458004170+00:00 stderr F Exception in thread "sbts_edr_dlr250__20240403T1024Z" java.lang.NoSuchMethodError: 'java.lang.Object io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.executeRules(java.lang.String, java.lang.String, org.apache.pinot.shaded.org.apache.kafka.common.header.Headers, java.lang.Object, io.confluent.kafka.schemaregistry.client.rest.entities.RuleMode, io.confluent.kafka.schemaregistry.ParsedSchema, io.confluent.kafka.schemaregistry.ParsedSchema, java.lang.Object)' 2024-04-03 12:26:01.61

Jackie-Jiang commented 3 months ago

This usually indicates a version conflict. How do you build the Pinot? Can you please check if you are using a different version of confluent Kafka?

cvinacuasonoc commented 2 months ago

yes, I think that is the case. Those kafka brokers are running an old version of Kafka: 2.4.1

cvinacuasonoc commented 2 months ago

I have moved the Kafka Topic to another Cluster running kafka_2.13-3.1.1 and the issue persists.

I have updated my Pinot deployment to "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba" on Kubernetes using the Helm attached.

This is the output of the "version" REST method:

{ "pinot-protobuf": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-kafka-2.0": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-avro": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-distribution": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-csv": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-s3": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-yammer": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-segment-uploader-default": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-batch-ingestion-standalone": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-confluent-avro": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-thrift": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-orc": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-clp-log": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-pulsar": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-azure": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-gcs": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-dropwizard": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-hdfs": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-adls": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-kinesis": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-json": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-minion-builtin-tasks": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-parquet": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba", "pinot-segment-writer-file-based": "1.2.0-SNAPSHOT-13673f11508f00ae35f5bb12f9cf97b6897ebfba" } pinot-dev-issue12785.txt

TharinduDG commented 2 months ago

I'm also getting this error when I build from master. 1.1 release branch works fine. 

java.lang.NoSuchMethodError: 'java.lang.Object io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.executeRules(java.lang.String, java.lang.String, org.apache.pinot.shaded.org.apache.kafka.common.header.Headers, java.lang.Object, io.confluent.kafka.schemaregistry.client.rest.entities.RuleMode, io.confluent.kafka.schemaregistry.ParsedSchema, io.confluent.kafka.schemaregistry.ParsedSchema, java.lang.Object)'
        at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer$DeserializationContext.read(AbstractKafkaAvroDeserializer.java:532)
        at io.confluent.kafka.serializers.AbstractKafkaAvroDeserializer.deserialize(AbstractKafkaAvroDeserializer.java:188)
        at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:108)
        at io.confluent.kafka.serializers.KafkaAvroDeserializer.deserialize(KafkaAvroDeserializer.java:103)
        at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:111)
        at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:121)
        at org.apache.pinot.plugin.inputformat.avro.confluent.KafkaConfluentSchemaRegistryAvroMessageDecoder.decode(KafkaConfluentSchemaRegistryAvroMessageDecoder.java:53)
        at org.apache.pinot.spi.stream.StreamDataDecoderImpl.decode(StreamDataDecoderImpl.java:54)
        at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.processStreamEvents(RealtimeSegmentDataManager.java:576)
        at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager.consumeLoop(RealtimeSegmentDataManager.java:466)
        at org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager$PartitionConsumer.run(RealtimeSegmentDataManager.java:697)
        at java.base/java.lang.Thread.run(Thread.java:1583)
Jackie-Jiang commented 2 months ago

Just noticed this class: org.apache.pinot.shaded.org.apache.kafka.common.header.Headers This means it is a shading issue. @xiangfu0 Can you help take a look? Will #12712 solve this?