Apache XTable (incubating) is a cross-table converter for lakehouse table formats that facilitates interoperability across data processing systems and query engines.
Addresses #412
Moves the metadata to a single json field in the target metadata to allow for easier evolution of the metadata that is tracked. Previously we would add a new field in metadata of the targets when something new needed to be tracked which can become a maintenance overhead.
Brief change log
Adds utils for to/from json on TableSyncMetadata
Updates users of TableSyncMetadata to read/write a single field
Adds a version to the metadata so we can later evolve the metadata and fail on older readers
ghost
commented
5 months ago
What is the purpose of the pull request
Addresses #412 Moves the metadata to a single json field in the target metadata to allow for easier evolution of the metadata that is tracked. Previously we would add a new field in metadata of the targets when something new needed to be tracked which can become a maintenance overhead.
Brief change log
Verify this pull request
Adds unit tests around the json (de)serialization