apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.62k stars 802 forks source link

MMap support for IPC files #6709

Open fabianmurariu opened 1 week ago

fabianmurariu commented 1 week ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. We are using polars-arrow and would prefer to move to arrow-rs as we are looking into using Datafusion as well

Describe the solution you'd like We are dependent on mmap support for IPC format

Describe alternatives you've considered keep using polars-arrow

Additional context It looks like Buffer and Bytes have the right functions and support for Custom dealocation

tustvold commented 1 week ago

The docs could definitely be improved, but the way to do this is to use FileDecoder with a buffer constructed from your mmap file - https://docs.rs/arrow-ipc/latest/arrow_ipc/reader/struct.FileDecoder.html

fabianmurariu commented 1 week ago

I'll have a look these days, if it works out I'll raise a PR