search

rilldata / rill

Rill is a tool for effortlessly transforming data sets into powerful, opinionated dashboards using SQL. BI-as-code.
https://www.rilldata.com
Apache License 2.0
1.73k stars 117 forks source link

Upgrade to DuckDB v0.10.3 and add delta extension #5078

Closed AndrewRTsao closed 5 months ago

AndrewRTsao commented 5 months ago

DuckDB v0.10.3 adds native support for reading delta lake format: https://duckdb.org/2024/06/10/delta.html

We should upgrade our internal duckdb to v0.10.3 and include the Delta extension in our binary so that we can read from delta lake files in object storage directly in Rill.

begelundmuller commented 5 months ago

Closing this issue because:

  1. We already have an issue for upgrading to DuckDB 1.0: https://github.com/rilldata/rill/issues/5052
  2. The delta extension is automatically downloaded and loaded by DuckDB when first used: https://duckdb.org/docs/extensions/delta. Since it's not a frequently used extension, it may not be worth bundling it directly in our binary. Feel free to reopen this issue if you think we should bundle it directly in our binary.

Note that there may be other features needed to use the DeltaLake extension with Rill for production uses, namely around setting credentials correctly for S3 (we don't currently use DuckDB's native credentials management features, but we should!).

AndrewRTsao commented 5 months ago

Created the following placeholder ticket so we can test (and document) this once we update duckdb: https://github.com/rilldata/FE-private-issues/issues/24