search

jimhester / rarrow

R bindings to apache arrow
32 stars 1 forks source link

Consider adding support batches #3

Open javierluraschi opened 6 years ago

javierluraschi commented 6 years ago

@jimhester I'm planning to step out of Arrow work for a bit, so I'll brain dump what I found here.

From looking at the Arrow sources, specially the Java ones, looks like column bindings would be helpful but not sufficient to interoperate with other platforms like JAVA which expects higher level objects. For instance, we would want support for /arrow/format/Message.fbs which supports column binding but also batching, etc.

I haven't identified which are the API entry points in the Arrow C API, but you can find an example of what a client would need to execute if we had an R API. Code is written in reticulate:

sparklyr/R/arrow_data.R#L45-L65 (mostly, pa$RecordBatch, pa$schema and pa$RecordBatchFileWriter).

In case you need this, there is some executable code in sparklyr as a PoC:

https://github.com/rstudio/sparklyr/pull/1611

Namanm1994 commented 6 years ago

@jimhester hey! Does this code provide us with functionality of reading a parquet file in R using arrow. ?? Or we would have to work on that ?

Thanks.