search

duckdb / duckdb_delta

DuckDB extension for Delta Lake
MIT License
139 stars 15 forks source link

write support #119

Open Ben-Epstein opened 1 week ago

Ben-Epstein commented 1 week ago

Hi folks, thanks for building such a great tool and extension.

I'm curious where write support falls on the priority list. I'm working on a data lakehouse+etl architecture and the write step is quite clunky as I've written it myself. I'm sure there are a lot of folks who are thinking the same thing :)

Is this something you have a general timeline for? Or even a best practices guide for getting data from duckdb into Delta (I'm using Polars but there may be a better option)

Thanks!

samansmink commented 1 week ago

Write support is definitely high on the prio list. The peeps over at the delta kernel (the library duckdb_delta heavily relies on) are working really hard on this as we speak! The plan for duckdb_delta is to keep up with the latest features of the delta kernel as quickly as possible. However, I can't give an exact timeline here

Ben-Epstein commented 4 days ago

Thanks @samansmink! Right now i'm using polars to write to my delta lake. Are they using something different for their write_delta functionality that isn't using the same delta kernel?

samansmink commented 3 days ago

@Ben-Epstein I believe they use https://github.com/delta-io/delta-rs