search

uwdata / arquero

Query processing and transformation of array-backed data tables.
https://idl.uw.edu/arquero
BSD 3-Clause "New" or "Revised" License
1.22k stars 64 forks source link

Adds the option for toArrowIPC to write to stream or file formats #299

Closed stuartlynn closed 1 year ago

stuartlynn commented 1 year ago

When serializing to IPC format using 

table.toArrowBuffer()

There is no way to specify if the serialization should be in IPC "stream" or "file" format.

The call to tableToIPC here:

https://github.com/uwdata/arquero/blob/074e1efdb6b8df2ca125db0c36020a8feb1d2d66/src/format/to-arrow.js#L6-L8

does not allow any passing of the second option defined in apache-arrow

https://github.com/apache/arrow/blob/5611f2bd0d6136b005d137a84b50709fc5c813bb/js/src/ipc/serialization.ts#L61-L65

This PR adds a {format} option to the options parameter of toArrowBuffer to allow specification of the IPC format

jheer commented 1 year ago

Thanks!