Prototype reading / writing to/from Apache arrow data structure

nwfsc-fram / OceanTS

Oceanographic data processing in Typescript using NodeJS and Apache Arrow

0 stars 3 forks source link

Prototype reading / writing to/from Apache arrow data structure #2

Open ghost opened 5 years ago

ghost commented 5 years ago

[x] Reading
[x] Appending columns to existing Apache Arrow Table
[ ] Writing

ghost commented 5 years ago

Able to open and read arrow data structure, still working on the best way to persist it in storage. Links of interest:

IPC Writer - https://issues.apache.org/jira/browse/ARROW-2116
https://github.com/apache/arrow/pull/2035

http://arrow.apache.org/docs/python/ipc.html#writing-and-reading-random-access-files https://arrow.apache.org/docs/ipc.html https://github.com/apache/arrow/issues/2119

ghost commented 5 years ago

I'm moving this back to the ToDo list for now, as the writing to an arrow file will be used once we get into the visual QA/QC work, which will happen after the initial bulk processing.