facebookincubator / velox

A C++ vectorized database acceleration library aimed to optimizing query engines and data processing systems.
https://velox-lib.io/
Apache License 2.0
3.27k stars 1.08k forks source link

Parquet support for timestamp data type #6527

Open chliang71 opened 9 months ago

chliang71 commented 9 months ago

Description

Currently the Parquet reader does not seem to support TIMESTAMP data type. We ran into exception at here, where it complains TypeKind::TIMESTAMP is not supported.

Then I found this draft PR to support INT96 timestamp. But we also have Parquet files with INT64 microseconds TIMESTAMP values. I have done some experimental changes internally to support this, but would like to see if there is any plans/advices/concerns/blockers in the community.

@xiaoxmeng Could you possibly provide some insight into this matter? Thanks!

CC @ying

mskapilks commented 8 months ago

Is the support for INT64 planned in future? Is someone working on this?

aditi-pandit commented 8 months ago

@yingsu00 @nmahadevuni