Updating the Variant and shredding specs from a thorough review.
What changes are included in this PR?
Spec updates, mostly to the shredding spec to minimize it and make it clear. This also attempts to make the variant spec more consistent (for example, by using value in both).
Removes object and array in favor of always using typed_value
Makes list element and object field groups required to avoid unnecessary null cases
Separates cases for primitives, arrays, and objects
Adds individual examples for primitives, arrays, and objects
Adds Variant to Parquet type mapping for shredded columns
Clarifies that metadata must be valid for all variant values without modification
Updates reconstruction algorithm to be more pythonic
Rationale for this change
Updating the Variant and shredding specs from a thorough review.
What changes are included in this PR?
Spec updates, mostly to the shredding spec to minimize it and make it clear. This also attempts to make the variant spec more consistent (for example, by using
value
in both).object
andarray
in favor of always usingtyped_value
required
to avoid unnecessary null casesmetadata
must be valid for all variant values without modificationDo these changes have PoC implementations?
No.