search

apache / parquet-java

Apache Parquet Java
https://parquet.apache.org/
Apache License 2.0
2.48k stars 1.37k forks source link

Is there any actual conversion implementation for arrow and parquet? #2928

Open chenyuanxing opened 1 week ago

chenyuanxing commented 1 week ago

We found that there are only schema conversions under the parqeut-arrow, so we wanted to ask if there is any code that includes the actual data conversions between parquet and arrow.

wgtmac commented 1 week ago

If possible to use C++, I think parquet-cpp in the Apache Arrow is the best solution to your case: https://arrow.apache.org/docs/cpp/parquet.html

chenyuanxing commented 6 days ago

Yes, We know there is a c++ implementation here, but I was wondering if there is a corresponding implementation for java, since all our code is java .

chenyuanxing commented 6 days ago

parquet-arrow The library looks like it's meant to do this, But I don't know why it's always just the schema part.

wgtmac commented 6 days ago

I think conversion between parquet and arrow is a valid use case. The parquet-java provides built-in row-level interfaces like avro/thrift/protobuf. Other parquet (Java) implementations (Presto/Trino/Spark) simply leverage the page & metadata reader/writer from this library to build extensions. Extending native arrow support would be a welcome extension to this library, IMO.

chenyuanxing commented 14 hours ago

So, the library parquet-arrow hasn't been used yet? because it only has schema mappings.

And We've looked at transformations in Spark, which are missing some types due to limitations in Spark, such as uint.So it's not really a universal conversion.