timestamps getting read in as `Int128`, `INT96` in the schema

JuliaIO / Parquet.jl

Julia implementation of Parquet columnar file format reader

Other

119 stars 32 forks source link

timestamps getting read in as `Int128`, `INT96` in the schema #35

Closed ExpandingMan closed 4 years ago

ExpandingMan commented 5 years ago

I'm seeing this happen to timestamps when reading in a snappy compressed parquet. These show up as normal timestamps (some are DateTime, others are Date) when read in Python using pyarrow.

Attempting to look into this now.

xiaodaigh commented 4 years ago

Hope it's not caused by the arrow schema

tanmaykm commented 4 years ago

Int96 timestamps were represented as Julia Int128. But with #50 and https://github.com/JuliaIO/Parquet.jl/pull/71/commits/d8c1739153e563f64831e250d5d58555ffcbc37a it should be possible to read them as Julia DateTime now.

Closing.