avro-kotlin / avro4k

Avro format support for Kotlin
Apache License 2.0
198 stars 37 forks source link

Not able to use @AvroName and @AvroFixed with dates and double during Serialization in pojo classes #231

Closed SamyakJ05 closed 3 months ago

SamyakJ05 commented 4 months ago

Describe the bug

When using the @AvroFixed and @AvroName property with DateTime and double, I am getting an error

Stack trace Error serialising Avro Message Caused by: java. lang.ClassCastExcept ion: value 10580 (a java. lang. Integer) cannot be cast to expected type lastDate at Emplyee. lastDate at org. apache.avro.path. TracingClassCastException.summarize (TracingClassCastException. java: 79) at org. apache.avro.path. TracingClassCastException.summarize (TracingClassCastEx ception. java: 30) at org. apache.avro.generic.genericDatumWriter.write(GenericDatumWriter. java: 84 at io.confluent.kafka.serial izers.AbstractKafkaAvroSerializer writeDatum (Abstr actKafkaAvroSerializer.java: 192) at io.confluent.kafka.serializers.AbstractKafkaAvroSerializer. serializeImpl(Ab stractKafkaAvroSerializer• java: 162)

To Reproduce Code to reproduce the behavior:

@AvroFixed(10) @AvroName("DATE") @Serializable(with = LocalDateSerializer::class) val lastDate : LocalDate?,

Expected behavior

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

Chuckame commented 4 months ago

Hello,

How is supposed to be the binary representation of a LocalDate ? What is this LocalDateSerializer, is it your own serializer ? Because natively avro4k is not able to convert an int (I suppose you want a sort of logical type date) to a fixed ByteArray of X bytes.

You need to provide your own serializer to be able of serializing a LocalDate or any other type to a fixed.

SamyakJ05 commented 4 months ago

It's not a binary representation of a LocalDate. They are two separate serializers, LocalDate and Double. I want to fix the size of LocalDate and Double using AvroFixed and name both of them using AvroName, which I am not able to. I don't want to convert it, just want to fix the size of both of these.

Chuckame commented 4 months ago

Please provide your custom serializers to understand. If no custom serializer, as I said, a LocalDate is represented by default as an int with the date logical type. An int is not a fixed type, so the AvroFixed annotation has no meaning. Same for double.

I don't want to convert it, just want to fix the size of both of these.

So if you do not want to convert the values to binary data, then I even less understand your point.

Please check the official docs to better understand how are working the standard primitives and fixed types: https://avro.apache.org/docs/1.11.1/specification/#primitive-types

SamyakJ05 commented 4 months ago

We are not using any custom serializer. Okay so date and double are represented as an int type by default and it is not possible to use @AvroFixed annotation with it.

My goal was to fix the amount of characters with date and double data type for which I was trying to use the AvroFixed annotation.

Thank You for you reply.

Chuckame commented 4 months ago

fix the amount of characters

But as it cannot be a fixed type, what is the purpose ? Do you want to serialize the date as an ISO string and the double as a string ?

The int is serialized following varint + zig-zag encoding, and the double following IEEE 754. You cannot change this encoding.

If you want to encode them as string, then you need to migrate to avro4k v2 and use @AvroStringable to convert them to the string type. But still, if you need to apply some custom logic, then create your own AvroSerializer (v2 also) or pre-map your expected fixed string content.

Chuckame commented 3 months ago

I'm closing this issue, as the main request is about a non-standard usage of avro. Please explain again and re-open the issue if the proposed solution doesn't fit to your needs.