Closed sauliusvl closed 1 year ago
See also this related issue: #4581
I think we just need to add a couple classes to convert string/bytes for UUIDs and update the builders to use them. @nastra is looking into this.
I opened https://github.com/apache/iceberg/pull/7399 to address this for Spark
This has been fixed by https://github.com/apache/iceberg/pull/7399 and also backported to Spark 3.3 + 3.2 with https://github.com/apache/iceberg/pull/7496 / https://github.com/apache/iceberg/pull/7497
hey nastra how can I use this fix https://github.com/apache/iceberg/pull/7399 using pyspark and aws glue?
hey nastra how can I use this fix #7399 using pyspark and aws glue?
@igniris87 make sure to use Iceberg 1.3.0+ as this is the version where the fix was shipped with
Consider a table created in Trino (it has a native UUID type that maps to UUID in Iceberg):
Everything looks good, I can query the table, running
parquet meta
on the resulting file in HDFS suggests it's written correctly according to the parquet specification (byte array of length 16 with logical typeUUID
):now I try to read it in Spark:
Same thing when trying to insert from spark:
The docs seem to suggest that UUID should be converted to a string in Spark, but after reading the source code I don't see how is this supposed to work: the
UUID
type gets simply mapped toString
here, which is not enough - the byte representation of a UUID can't be cast toString
straightforwardly.The way I see it Iceberg should either:
java.util.UUID
if they really want to see the text representation - optimal, but not user friendly, also not sure how to propagate the logical type to parquet