Implement Apache Arrow methods

heavyai / heavyai.jl

Julia client for OmniSci GPU-accelerated SQL engine and analytics platform

https://omnisci.github.io/OmniSci.jl/latest/

Other

21 stars 9 forks source link

Implement Apache Arrow methods #32

Open randyzwitch opened 5 years ago

randyzwitch commented 5 years ago

Implement remaining load_table methods, with an eye on removing unneeded methods (such as row-wise loading instead of columnar)

~load_table_binary(conn::OmniSciConnection, table_name::String, rows::Vector{TRow})~

[x] load_table_binary_columnar(conn::OmniSciConnection, table_name::String, cols::Vector{TColumn})
[ ] load_table_binary_arrow(conn::OmniSciConnection, table_name::String, arrow_stream::Vector{UInt8})
[ ] sql_execute_df
[ ] sql_execute_gdf

randyzwitch commented 5 years ago

Removing load_table_binary from focus, as there is an error that I currently don't understand. It appears that pymapd doesn't implement the method, which might give a good indication its not needed

randyzwitch commented 5 years ago

Remaining method needs Arrow support

randyzwitch commented 5 years ago

This isn't just creating Arrow vectors (from Arrow.jl), but a stream/recordbatch. Might need to use ArrowGlib.jl

randyzwitch commented 5 years ago

WIth https://github.com/JuliaData/FlatBuffers.jl/issues/24, another piece for Arrow support getting closer. With Flatbuffer flatc, should be able to move closer to a native Arrow implementation to read recordbatches