search

pola-rs / polars-cli

CLI interface for running SQL queries with Polars as backend
https://pola.rs/
MIT License
160 stars 12 forks source link

Add support for `read_delta` #14

Open MrPowers opened 1 year ago

MrPowers commented 1 year ago

Problem description

The Polars CLI supports read_csv and read_parquet, but it doesn't look like read_delta is supported yet.

Polars supports read_delta, so I am hoping it won't be too hard to add it to the Polars CLI.

Screenshot 2023-07-20 at 10 37 17 AM
stinodego commented 1 year ago

read_delta is implemented in Python, not on the Rust side. So the CLI does not have access to it.

I would be in favor of adding this, but it's not an easy one. I believe @ritchie46 mentioned at one point that he preferred to have the read_delta functionality on the Python side only, but I'm not really sure why (maybe I am misremembering).

The dependency we use for read_delta in Python is implemented in Rust, so it's probably possible to have this on the Rust side.

ritchie46 commented 1 year ago

Delta-rs uses arrow-rs, so we would need to compile a whole new arrow implementation. We could add it under a separate crate that users could draw in if they want to pay the compilation cost.

For the CLI it can be fine if we have binaries.