Support for delta

[Yacobolo]

Looking forward to the support for delta. This would enable us to run a poor man's data lakehouse! Do you need any help? What is the eta - this year?

[jwills]

Ack, sorry for the lag here @Yacobolo, I was on the road and missed this going by. I would like to have a plugin that supported Delta akin to the one I have for Iceberg; I'm assuming it would use the deltalake python package, but I personally don't have access to a Delta lake instance and tbh don't really care enough about learning how to setup a real one to do it myself "for fun."

However, if you (or anyone else!) does have a Delta lake instance and you know it should be configured as a dbt-duckdb plugin, I would most definitely be happy to merge it in.

[milicevica23]

Hi, @jwills, I would like to try this integration. This would be my first contribution, so I would appreciate some help and guidance at the beginning.

I did a first draft of read plugin integration here
and doing parallel an example project here where i showcase it

Here is the source configuration which loads data as the source with file and projection prunning

What workflow works best for you that you are able to give a feedback?

[jwills]

Hey @milicevica23, thanks so much for taking a crack at this!

The code as-written makes sense to me, but I have to be honest that I don't have a great sense for how folks actually use the deltalake python module in the real world-- like, do folks really use delta tables w/o a catalog? https://delta-io.github.io/delta-rs/python/usage.html#loading-a-delta-table

[milicevica23]

The nice thing is that you can but should not use a catalog to know where your table is and i thought to implement support for both ways. Or at least try to do it.. You can think of that as that we add a new file format to external files and not everybody who is on prem or doing simple projects have catalogs. But would be happy to hear feedback from others

[Yacobolo]

Same here, the main use case is not the catalog, but more the metadata it generates together with the ACID transactions and time travel / change history🔥

[jwills]

Alright, super cool. So @milicevica23 if you would put your change together as a PR and other folks on this thread can weigh in on any additional config options we need to support those use cases, that would be great!

[milicevica23]

Sure, i will open an draft PR.

The things still to do

• check/rename naming for config params (feedback appreciate)
• add support for time traveling to load older versions of the tables
• add support for at least azure because i have access to, if you have some guidance for AWS s3 local environment like Lolcalstack i can try that too but have to learn it too.
• (optional) try to rewrite the reading logic that we don't have to import first and then process but that we can push down the predicates. I made an stack-overflow question, maybe you can help me to do/uderstand that @jwills? The benefit will be that we don't have to specify filters on the config level but it will come from the first query on the view which is pointing/ holding an instance to the delta table
• (optional) try to add databricks catalog option. I don't have so much experience with that but can learn it on the way
• do testing after we agree on the structure

Be free to add new ideas, topics

I am not used to PR process in the github so feel free to rewrite, do stuff as it fits the needs and best practices

[geoHeil]

How would https://duckdb.org/2024/06/10/delta.html the new delta kernel work here to simplify and perhaps make the access to delta based data more performant?

[geoHeil]

A: https://duckdb.org/docs/extensions/delta#supported-duckdb-versions-and-platforms simply adding the extension (if the platform is supported)