Open wjones127 opened 1 year ago
Has there been any progress towards this? I would be very interested in seeing what I could do to contribute here!
Hi @DOH-WLD0303, nice to hear!
I'm currently working upstream on the foundations on https://github.com/apache/arrow-rs/pull/3582
Once, that's done, I'll finish up my work in #945. And then once that is done more of these issues will be available. It may take me a few weeks to finish those though.
In the meantime, if you want you would work in parallel on implementing or improving some of the underlying operations in https://github.com/delta-io/delta-rs/tree/main/rust/src/operations, that would be helpful as well. I think OPTIMIZE, DELETE, and UPDATE are ones we still haven't done.
@wjones127 , sincerest apologies but my work timeline isn't going to give me the bandwidth to assist on this for right now. I've never dipped my toes into rust either which is definitely a barrier for me unfortunately. I'm hoping I can learn enough at some point to contribute a little!
Description
I'm thinking about something we might want to work toward in 2023, and an ADBC driver might be a good focus. We would aim to use this to support integration with DuckDB and Polars, as well as to upgrade the Python reader and writer. (DataFusion could also use this, but likely we will have direct support using the underlying implementation; see the doc for details.)
Draft design doc here, open to comments: https://docs.google.com/document/d/1ud-iBPg8VVz2N3HxySz9qbrffw6a9I7TiGZJ2MBs7ZE/edit?usp=sharing
Use Case
Make it easy to build connectors to Delta-rs, no matter the language, and with strong guarantees of implementation correctness.
Related Issue(s)
Work on ADBC driver (in rough priority order):
SELECT
queriesSELECT COUNT(*)
queriesOPTIMIZE
andVACUUM
CREATE
andINSERT INTO/OVERWRITE
UPDATE
DELETE
MERGE
Downstream work enabled by driver: