API: Add compatibility checks for Schemas with default values

apache / iceberg

Apache Iceberg

https://iceberg.apache.org/

Apache License 2.0

6.51k stars 2.25k forks source link

Closed rdblue closed 3 weeks ago

rdblue commented 3 weeks ago

This updates Schema.checkCompatibility:

Adds a check that initial-default is not set for v1 and v2 tables, which would break forward compatibility
Updates the method to return a list of compatibility problems rather than one at a time
Adds tests for checkCompatibility in TestSchema

I think that this should get into 1.7.0 to avoid accidentally leaking initial defaults in v1 and v2 tables.

RussellSpitzer commented 3 weeks ago

Tests failing are because Metadata.json also has validations for this and this check is failing first

rdblue commented 3 weeks ago

Updated. Thanks, @RussellSpitzer!

RussellSpitzer commented 3 weeks ago

Thanks everyone - @rdblue (for original fix), @Fokko , @anuragmantri , @kevinjqliu , @singhpk234 , @amogh-jahagirdar For reviews!