delta-io / delta-rs

A native Rust library for Delta Lake, with bindings into Python
https://delta-io.github.io/delta-rs/
Apache License 2.0
1.97k stars 365 forks source link

Support for SQL syntax for filtering similar to PySpark #2625

Open Josh-Hiz opened 3 days ago

Josh-Hiz commented 3 days ago

Description

Would it be possible to support SQL strings to query deltalake from whatever source you desire? Similar to what PySpark does when you perform a read operation you can apply a full SQL string as a filter/

Use Case

sql_str = """
SELECT your_column
FROM your_table
ORDER BY your_column DESC
LIMIT 10
"""
delta_url = "..."
storage_options = {...}
dl: DeltaTable = DeltaTable(table_uri=delta_url, storage_options=storage_options)
df = dl.to_pandas(filters = sql_str)

Or something at the very least similar to this?