avro-kotlin / avro4k

Avro format support for Kotlin
Apache License 2.0
188 stars 36 forks source link

Question: Why is single object encoding not supported? #138

Closed jangalinski closed 1 year ago

jangalinski commented 1 year ago

We are using single object encoding with the java SpecificRecordBase classes generated by the maven avro plugin.

I would love to switch to kotlin data classes, using avro4k, bit the README states that The single object encoding is currently not supported.

I would like to understand why this is the case, it seems that you are able to encode/decode to bytearray, so using singleobject encoding does not seem to far fetched. Did you already try to implement this and ran into unsolvable problems? Or would this be a feature that could rather easily be contributed? Interested in your opinion end experience before I start trying and waste time.

thake commented 1 year ago

Hey @jangalinski, thanks for your question. Single object encoding has just not been implemented. We tried to implement all encodings that are available out of the box with Avro. Do you mind giving a code example of how you encode your object using single object encoding in Java? I've scanned through the avro java library but couldn't find any official encoder for this format.

jangalinski commented 1 year ago

Currently I use generated java classes as messages, the single object encoding is built in by the maven avro generator.

If it is possible to convert a serializable data class to a generic record, the record itself than can be encoded to single object using:

fun encode(data: GenericData.Record) = ByteArrayOutputStream().use {
    BinaryMessageEncoder<GenericData.Record>(GenericData.get(), data.schema).encode(data, it)
    return it.toByteArray()
  }
jangalinski commented 1 year ago

I opened feature request #141 and will try to work on a first PoC asap.

jangalinski commented 1 year ago

Do you mind giving a code example of how you encode your object using single object encoding in Java? I've scanned through the avro java library but couldn't find any official encoder for this format.

As you can see in #142: BinaryMessageEncoder and BinaryMessageDecoder are the classes that provide this. Both can work on streams and GenericRecords so it was not too hard to implement this. Looking forward for feedback ...

jangalinski commented 1 year ago

closing this as this is a) answered and b) PR is pending