search

kestra-io / plugin-kafka

https://kestra.io/plugins/plugin-kafka/
Apache License 2.0
4 stars 5 forks source link

fix: handle complex types when generating Avro values #94

Closed yvrng closed 1 month ago

yvrng commented 1 month ago

When using the Produce task, we are only able to send objects with a simple structure. The current code does not handle complex types correctly, such as enums, unions, lists, maps, etc.

This commit improves handling of complex Avro types in Kafka by introducing MapToGenericRecordSerializer a better GenericRecord serialization.

Note: Further work may be required to handle fixed values (which only accept byte[]) and unions (currently using the first non-null type, which allow support for optional values but not polymorphism).

loicmathieu commented 1 month ago

Can you add a new tests that didn't pass without this change?

yvrng commented 1 month ago

I've added a collection of unit tests for all complex types, and for some common combinations.