search

findify / flink-adt

Scala ADT support for Apache Flink
MIT License
51 stars 9 forks source link

State schema evolution question #1

Closed izeigerman closed 3 years ago

izeigerman commented 4 years ago

Hey @shuttie. Thank you for this project! I'm currently considering to use your library but there is one detail that is important to me and which is not well covered in the README. How well generated serializers handle schema evolution for product types? Is it more like Flink's CaseClassSerializer (no schema evolution support), or like PojoSerializer? To give you more context - I care only about field removal and addition (no type changes). Thank you!

shuttie commented 3 years ago

I've added a note in the doc about schema evolution: https://github.com/findify/flink-adt#schema-evolution

maxjoh commented 1 year ago

Hi @shuttie, I would like to question the updated documentation about this. As @izeigerman writes, Flink's CaseClassSerializer does not have schema evolution support.

Only POJO and Avro is supported https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#supported-data-types-for-schema-evolution

Am I correct or does flink-adt actually have schema evolution support for scala case classes?