search

apache / parquet-java

Apache Parquet Java
https://parquet.apache.org/
Apache License 2.0
2.65k stars 1.41k forks source link

Is there any actual conversion implementation for arrow and parquet? #2928

Open chenyuanxing opened 5 months ago

chenyuanxing commented 5 months ago

We found that there are only schema conversions under the parqeut-arrow, so we wanted to ask if there is any code that includes the actual data conversions between parquet and arrow.

wgtmac commented 5 months ago

If possible to use C++, I think parquet-cpp in the Apache Arrow is the best solution to your case: https://arrow.apache.org/docs/cpp/parquet.html

chenyuanxing commented 4 months ago

Yes, We know there is a c++ implementation here, but I was wondering if there is a corresponding implementation for java, since all our code is java .

chenyuanxing commented 4 months ago

parquet-arrow The library looks like it's meant to do this, But I don't know why it's always just the schema part.

wgtmac commented 4 months ago

I think conversion between parquet and arrow is a valid use case. The parquet-java provides built-in row-level interfaces like avro/thrift/protobuf. Other parquet (Java) implementations (Presto/Trino/Spark) simply leverage the page & metadata reader/writer from this library to build extensions. Extending native arrow support would be a welcome extension to this library, IMO.

chenyuanxing commented 4 months ago

So, the library parquet-arrow hasn't been used yet? because it only has schema mappings.

And We've looked at transformations in Spark, which are missing some types due to limitations in Spark, such as uint.So it's not really a universal conversion.

doki23 commented 3 months ago

It seems that iceberg has an arrow implementation.

wgtmac commented 3 months ago

It seems that iceberg has an arrow implementation.

Yes, but it does not support reading repetition levels and v2 encodings.

doki23 commented 2 months ago

I've ported Iceberg's implementation to parquet-arrow and removed the concepts specific to Iceberg. If someone is interested in this, they are welcome to make contributions.