search

apache / paimon

Apache Paimon is a lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations.
https://paimon.apache.org/
Apache License 2.0
2.39k stars 942 forks source link

[Bug] Timestamp LTZ is unsupported in table store #734

Closed JingsongLi closed 1 year ago

JingsongLi commented 1 year ago

Search before asking

Paimon version

0.4

Compute Engine

flink

Minimal reproduce step

run table with timestamp with local zone field.

What doesn't meet your expectations?

Due to orc format limitation, timestamp ltz is unsupported now. We should fix this, and validate this type cross multiple engines (hive spark trino). We need to careful about time zone.

Anything else?

No response

Are you willing to submit a PR?

hk-lrzy commented 1 year ago

@JingsongLi can you assign this task to me, i am interest on this task.

JingsongLi commented 1 year ago

@JingsongLi can you assign this task to me, i am interest on this it.

Orc has been fixed, we need to fix spark, hive, trino.. And parquet.

JingsongLi commented 1 year ago

@hk-lrzy assigned to you.

hk-lrzy commented 1 year ago

@JingsongLi can you assign this task to me, i am interest on this it.

Orc has been fixed, we need to fix spark, hive, trino.. And parquet.

i think you mean avro， i see Parquet already supported, follow is parquet's code:

            case TIMESTAMP_WITH_LOCAL_TIME_ZONE:
                LocalZonedTimestampType localZonedTimestampType = (LocalZonedTimestampType) type;
                return localZonedTimestampType.getPrecision() <= 6
                        ? Types.primitive(INT64, repetition).named(name)
                        : Types.primitive(PrimitiveType.PrimitiveTypeName.INT96, repetition)
                                .named(name);
legendtkl commented 1 year ago

hi, I would like to take this task if it's suspend.