search

julianpeeters / avrohugger

Generate Scala case class definitions from Avro schemas
Apache License 2.0
201 stars 120 forks source link

Constraints on using Customizable Namespace Mapping #186

Open rsitze-mmai opened 10 months ago

rsitze-mmai commented 10 months ago

These seem to work fine with a Kafka Producer, not so well for a Kafka Consumer: the confluent KafkaAvroDeserializer is looking for (package name, class name) matching the names in the Schema - which aren't there.

I'm not finding any clear/specific documentation for that tooling that would bridge the customized namespaces into that world. This is all new to me, so at the least, a pointer in the README to where to go next? Perhaps this just doesn't work?

julianpeeters commented 10 months ago

thanks for the report

Perhaps this just doesn't work?

unknown. possibly running into warning 1. if I were in your shoes, I might lean towards generating regular java classes (using sbt-avro or avro-tools), which are expected to be fully compatible with the avro ecosystem.

otherwise, if you were determined to investigate and/or get this to work, I'd accept a fix or ideas for integration tests could go here

rsitze-mmai commented 10 months ago

Warning 1 (private fields) isn't the immediate problem - I think it really is as simple as the runtime deserializer uses the schema namespace + schema's record name as package + class name, and can't find it. That was pretty much the error I was getting anyway.

On the second point, I don't see how this can be fixed. I did look at the deserializer properties / options, and nothing really jumps out to me screaming "use THIS to map a schema type to a fully qualified classname". I poked around the code a bit, but I'm just not familiar with all the in/outs of the confluence apis/implementation.

To be clear - the consumer works just fine as long as I don't customize the namespace mapping - and that's good enough for me - so Thank You for avrohugger.

At this point, a simple statement on namespace mappings might be added: "Namespace mappings work for with KafkaAvroSerializer but not for KafkaAvroDeserializer; if anyone knows how to configure the deserializer to map incoming schema names to target class names please speak up!"

julianpeeters commented 10 months ago

"Namespace mappings work for with KafkaAvroSerializer but not for KafkaAvroDeserializer; if anyone knows how to configure the deserializer to map incoming schema names to target class names please speak up!"

good stuff, thanks