snowplow-incubator / common-streams

Other
1 stars 0 forks source link

Preserve original field order when merging schemas #98

Closed istreeter closed 6 days ago

istreeter commented 2 weeks ago

See snowplow/schema-ddl#213

When a schema is evolved (e.g. from 1-0-0 to 1-0-1) we create a merged struct column combining fields from new and old schema.

For some loaders it is important that newly-added nested fields come after the original fields. E.g. Lake Loader with Hudi and Glue sync enabled.