search

apache / pulsar

Apache Pulsar - distributed pub-sub messaging system
https://pulsar.apache.org/
Apache License 2.0
14.23k stars 3.58k forks source link

Avro "fixed" field causes ClassCastException #9718

Open patricklucas opened 3 years ago

patricklucas commented 3 years ago

Describe the bug If you have an Avro "fixed" field in a record (or presumably if you try to produce a "fixed" field directly), you get a ClassCastException when producing a message.

java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app')

(full traceback below)

I slightly tweaked @sijie 's pulsar-avro-schema-example project to demonstrate the issue: patricklucas/pulsar-avro-schema-example.

(I updated dependencies, and then added a fixed field to the TwitterSchema record. Additionally, I added code to serialize the message to JSON before producing, to demonstrate that Avro itself handles the message correctly)

It's possible this is unsupported in some way by Pulsar, but I couldn't find anything in the documentation about such limitations, so I thought I'd ask here.

To Reproduce Steps to reproduce the behavior:

  1. Clone the project patricklucas/pulsar-avro-schema-example
  2. Run TweetProducer, passing a Pulsar service URL as a single argument
  3. Observe error

Expected behavior The message should be serialized successfully.

Additional context

Full traceback:

org.apache.pulsar.client.api.SchemaSerializationException: java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app')
    at org.apache.pulsar.client.impl.schema.writer.AvroWriter.write(AvroWriter.java:55)
    at org.apache.pulsar.client.impl.schema.AbstractStructSchema.encode(AbstractStructSchema.java:50)
    at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.value(TypedMessageBuilderImpl.java:178)
    at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:63)
    at org.apache.pulsar.examples.TweetProducer.main(TweetProducer.java:43)
Caused by: java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app')
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeFixed(GenericDatumWriter.java:339)
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:147)
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
    at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:206)
    at org.apache.pulsar.shade.org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:101)
    at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:186)
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:195)
    at org.apache.pulsar.shade.org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:130)
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:82)
    at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
    at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
    at org.apache.pulsar.client.impl.schema.writer.AvroWriter.write(AvroWriter.java:53)
    ... 4 more
codelipenghui commented 2 years ago

The issue had no activity for 30 days, mark with Stale label.

patricklucas commented 2 years ago

This still occurs on latest Pulsar.

I've updated the repo with the reproducing example to upgrade to Pulsar 2.9.1 and to make it easier to run:

Full output:

$ docker-compose up -d
[+] Running 2/2
 ⠿ Network pulsar-avro-schema-example_default     Created                                                                                                                                                                                                                              0.0s
 ⠿ Container pulsar-avro-schema-example-pulsar-1  Started 
$ ./gradlew :runTweetProducer

> Task :runTweetProducer
...
Tweet JSON: {
  "username" : "user-0",
  "tweet" : "tweet-0",
  "fixed" : "Hello",
  "timestamp" : 1646385474269
}
org.apache.pulsar.client.api.SchemaSerializationException: java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app') in field fixed
        at org.apache.pulsar.client.impl.schema.writer.AvroWriter.write(AvroWriter.java:55)
        at org.apache.pulsar.client.impl.schema.AbstractStructSchema.encode(AbstractStructSchema.java:57)
        at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.value(TypedMessageBuilderImpl.java:175)
        at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:64)
        at org.apache.pulsar.examples.TweetProducer.main(TweetProducer.java:42)
Caused by: java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app') in field fixed
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.addClassCastMsg(GenericDatumWriter.java:191)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:229)
        at org.apache.pulsar.shade.org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:101)
        at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:184)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210)
        at org.apache.pulsar.shade.org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
        at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
        at org.apache.pulsar.client.impl.schema.writer.AvroWriter.write(AvroWriter.java:53)
        ... 4 more
Caused by: java.lang.ClassCastException: class org.apache.pulsar.examples.Fixed cannot be cast to class org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed (org.apache.pulsar.examples.Fixed and org.apache.pulsar.shade.org.apache.avro.generic.GenericFixed are in unnamed module of loader 'app')
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeFixed(GenericDatumWriter.java:362)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:148)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
        at org.apache.pulsar.shade.org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
        at org.apache.pulsar.shade.org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221)
        ... 13 more

BUILD SUCCESSFUL in 6s
4 actionable tasks: 1 executed, 3 up-to-date
SaraMahdy commented 1 year ago

This is because Pulsar-client uses a shade dependency for Apache-Avro. I faced a very similar problem and fixed it by using pulsar-client-original as a dependency instead of pulsar-client and it works fine for me. It's not obvious in Pulsar docs that there are client options to choose from, I had raised an issue about this a while ago: https://github.com/apache/pulsar/issues/21230