Closed FANNG1 closed 2 days ago
Shall we fix this in 0.6.0?
The problem still exists if using origin spark Iceberg connector, cc @jerryshao @diqiu50
I see. We can defer this issue to the next release.
Trino's default timestamp
precision is milliseconds. The timestamp type in Graviton does not handle precision. When using the timestamp
type, Trino does not know the precision of the type by default, which may cause problems in reading.
@mchades The timestamp
and TimeType
type in Graviton need to support precision.
is there other way to resolve this? I'm not sure if this is the right way .
We need to first determine what the problem is. The type of timestamp
in iceberg is second or millisecond or microsecond.
Timestamp is transformed to parquet TIMESTAMPTZ_MICROS in https://github.com/apache/iceberg/blob/main/parquet/src/main/java/org/apache/iceberg/parquet/TypeToMessageType.java#L138-L143
case TIMESTAMP:
if (((TimestampType) primitive).shouldAdjustToUTC()) {
return Types.primitive(INT64, repetition).as(TIMESTAMPTZ_MICROS).id(id).named(name);
} else {
return Types.primitive(INT64, repetition).as(TIMESTAMP_MICROS).id(id).named(name);
}
Version
main branch
Describe what's wrong
trino couldn't read Iceberg partition table created by spark
Error message and/or stacktrace
Query 20240828_134434_01832_9eicb failed: Could not serialize column 'hire_date' of type 'timestamp(3)' at position 1:4
How to reproduce
Spark sql:
trino:
Additional context
No response