An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
Spark master introduced collated StringType. In TypeWideningMetadata.collectTypeChanges, we used to only match StringType instance, therefore if the type change is from/to StringType(collationId), which is expected and unrelated to type widening, we will throw an error typeWidening.unexpectedTypeChange. This PR fixes this by matching both collated and uncollated StringType when capturing string type changes.
How was this patch tested?
Existent tests.
Tests related to collations cannot be added as delta does not support collations yet.
Which Delta project/connector is this regarding?
Description
Spark master introduced collated
StringType. InTypeWideningMetadata.collectTypeChanges, we used to only match StringType instance, therefore if the type change is from/toStringType(collationId), which is expected and unrelated to type widening, we will throw an errortypeWidening.unexpectedTypeChange. This PR fixes this by matching both collated and uncollatedStringTypewhen capturing string type changes.How was this patch tested?
Existent tests.
Tests related to collations cannot be added as delta does not support collations yet.
Does this PR introduce any user-facing changes?
No.