Closed AshinGau closed 4 months ago
Is this because trino, doris and arrow are assuming local timezone for timestamp values?
After detailed reading of the document definition of isAdjustedToUTC=true, I found that is was a display problem.
The top first image shows that the Arrow result is
2023-09-17 05:14:35.142+00:00
, which contains the time zone(relative to UTC0), it equals to 2023-09-17 13:14:35.142
in my local time zone(Asia/Shanghai UTC8), so although the display is different, the result is correct for Arrow because as stated in the Parquet document:
In practice, such timestamps are typically displayed to users in their local time zones, therefore they may be displayed differently depending on the execution environment.
However, the display of Trino does not include a time zone, and when trying to set different time zones, Trino still returns the same result, so it is highly likely that there is an issue with trino's results:
Describe the problem you faced I am a committer of Doris. When I use Doris to read the parquet file created by hudi, I find that the output of timestamp type is decreased by 8 hours. Then I use other tools(arrow, trino, spark) to check the result, it seems that different system parse different time zone of timestamp type:
The results of arrow, Doris, trino are the same, while the results of hudi, spark-shell are increased by 8 hours(local time zone is Asia/Shanghai).
To Reproduce Spark 3.3 + Hudi 0.14.1
Expected behavior Maybe it's right to output
2023-09-17 13:14:35.142
, because string liketimestamp'2023-09-17 13:14:35.142'
is inserted. However the parquet file create by hudi indicates that the timestamp type should be adjusted to UTC:The meta data shows that the timestamp is relative to UTC0, not the local time zone(UTC8), according to the definition of isAdjustedToUTC=true![image](https://github.com/apache/hudi/assets/19337507/1f1139a6-f58f-4026-8d9c-d828dc81779b)
Environment Description
Hudi version : 0.14.1
Spark version : 3.3.2
Hive version : 3.1.1
Hadoop version : 3.2.2
Storage (HDFS/S3/GCS..) :
Running on Docker? (no) :