[R] Support read and write orc file format

apache / arrow

Apache Arrow is the universal columnar format and multi-language toolbox for fast data interchange and in-memory analytics

https://arrow.apache.org/

Apache License 2.0

14.59k stars 3.54k forks source link

[R] Support read and write orc file format #24269

Open asfimport opened 4 years ago

asfimport commented 4 years ago

Currently the R package can read/write arrow, feather, parquet etc ... How feasible is it for orc file format to be support with read / write capabilities?

Reporter: Dyfan Jones

_{Note: This issue was originally created as ARROW-8056. Please see the migration documentation for further details.}

asfimport commented 4 years ago

Neal Richardson / @nealrichardson: Codewise, it probably wouldn't be too bad to add R bindings to cpp/src/arrow/adapters/orc, and it would probably look a lot like the parquet bindings. One bit of complexity is that you'd need to make it conditionally available based on whether Arrow C++ was built with ORC support enabled. AFAICT ORC is not available on Windows.

The bigger challenge (or at least effort required) would probably be the C++ dependency building, if we wanted this to be a feature generally available to all (non-Windows) users.

asfimport commented 4 years ago

Dyfan Jones: To my knowledge R doesn't have any maintained packages that create orc files without the help of spark. There is one package https://github.com/vertica/r-dataconnector that does create R data.frames to orc, however it doesn't appear to be actively maintained.

To give abit of context around this feature request. I am currently developing two R packages (https://github.com/DyfanJones/RAthena and https://github.com/DyfanJones/noctua) that connect to AWS Athena. arrow is used to upload parquet files to AWS S3 and then registered with AWS Athena. I would like to the same with orc files without having to spin up a local spark cluster.

asfimport commented 4 years ago

Neal Richardson / @nealrichardson: Sounds like a reasonable objective.

Since your packages are already using reticulate, I checked to see if pyarrow wheels are built with ORC support, and it appears they are not currently due to an issue with protobuf: https://github.com/apache/arrow/pull/5627

So, yeah, sadly the biggest challenge would be in C++ build support.

What is the advantage of using ORC over parquet in Athena?

asfimport commented 4 years ago

Dyfan Jones: Thanks for checking that out for me.

AWS actually doesn't recommend one over the other, it just says to use orc or parquet (https://aws.amazon.com/blogs/big-data/top-10-performance-tuning-tips-for-amazon-athena/)

For Athena, we recommend using either Apache Parquet or Apache ORC, which compress data by default and are splittable.

For the time being I am happy to push users to parquet as this is fully supported in the current version of arrow. :)

asfimport commented 4 years ago

Neal Richardson / @nealrichardson: Sounds good. For the future, we can leave this open and see if more ORC advocates turn up.