Open 0xgeert opened 2 years ago
The IPC readers are implemented upstream. Could you make this issue here? https://github.com/jorgecarleitao/arrow2
I am a bit surprised about pl.readIPC(fromArrow)
and pl.readIPC(fromIPC)
: shouldn't these be two different signatures? One thing is to read a stream (.ipc
), the other is a file (.arrow
). I think that we are just missing a readIPCStream
in Polars' API that can read arrow streams (as opposed to arrow files).
Ah.. Polars doesn't have that distinction no. So the IPC
is the stream and the .arrow
is the feather
file as the IPC data + additional headers?
Then we must add this.
Hi!
I'm keen to get this into polars, as Snowflake uses this for their response format and would be awesome to get it in for reading data straight from SF into Polars.
Here is a quick primer about the streaming files from Arrow: https://arrow.apache.org/docs/python/ipc.html And the guide here from arrow2 about reading the stream: https://jorgecarleitao.github.io/arrow2/io/ipc_stream_read.html
IMHO, supporting files initially is fine, later can do other streaming support.
I've started looking into this, and the major blocker I can see is projections.
In arrow2, projections are not supported here: https://github.com/jorgecarleitao/arrow2/blob/main/src/io/ipc/read/stream.rs#L185
So we will need to build the projection from the chunks.
Thoughts?
Transfering this to the NodeJS repo as I have no way to reproduce this using Python/Rust. Not sure if this is still relevant.
Using Node.JS
What version of polars are you using?
"nodejs-polars": "^0.2.0"
What operating system are you using polars on?
MacOS Big Sur 11.1
Describe your bug.
Reading in a buffer from an
.ipc
(ArrowStream) file usingreadIPC
fails withError: Arrow file does not contain correct header
. At the same time the file is not corrupt since it can be loaded using apache-arrow'sTable.from
methodWhat are the steps to reproduce the behavior?
See code example below. I'll post both the .arrow file (works) and .ipc file (doesn't work) as attachment