brimdata / zed

A novel data lake based on super-structured data
https://zed.brimdata.io/
BSD 3-Clause "New" or "Revised" License
1.34k stars 67 forks source link

Apache Arrow support #4226

Open philrz opened 1 year ago

philrz commented 1 year ago

Add support for Apache Arrow as another format the Zed tooling can read/write.

A bonus found during research is that the Arrow library we intend to use comes with a better Parquet implementation than the one we're using currently, so once we're making use of that some of our open Parquet issues may be addressed.

nwt commented 1 year ago

4252 added support for the Arrow IPC stream format. We can close this after we add support for the IPC file format.

nwt commented 1 year ago

4278 covers using the Arrow Parquet implementation.