An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Currently, SchemaMergingUtils.mergeSchemas will not widen fields when both keepExistingTypes and allowTypeWidening are set to true. This change will allow types to be widened when both keepExistingTypes and allowTypeWidening are true, and in the case of non-widening type changes, the existing type will be kept. This change can be used to fall back to existing types instead of throwing a DeltaAnalysisException when we want to merge a schema with widened fields that may also have non-widenable type changes.
Which Delta project/connector is this regarding?
Description
Currently,
SchemaMergingUtils.mergeSchemaswill not widen fields when bothkeepExistingTypesandallowTypeWideningare set to true. This change will allow types to be widened when bothkeepExistingTypesandallowTypeWideningare true, and in the case of non-widening type changes, the existing type will be kept. This change can be used to fall back to existing types instead of throwing aDeltaAnalysisExceptionwhen we want to merge a schema with widened fields that may also have non-widenable type changes.How was this patch tested?
New unit test in
SchemaUtilsSuite.scalaDoes this PR introduce any user-facing changes?
No