Iceberg Support Roadmap

Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust

https://getdaft.io

Apache License 2.0

2.34k stars 164 forks source link

Iceberg Support Roadmap #2458

Open kevinzwang opened 4 months ago

kevinzwang commented 4 months ago

Features

[x] Basic reads
[x] Partitioned reads
[x] Basic writes
[x] Merge-on-read position delete reads
[ ] Merge-on-read equality delete reads
- blocked by pyiceberg
[ ] Utilizing statistics from table format
[ ] #1953
[x] Field IDs
[ ] https://github.com/Eventual-Inc/Daft/issues/2105
[ ] https://github.com/Eventual-Inc/Daft/issues/2249
[x] Partitioned writes
[ ] Upserts/MERGE
- [ ] Copy-on-write writing
- [ ] Merge-on-read writing
- blocked by pyiceberg
[ ] Add support for writing new Iceberg tables to catalogs such as AWS Glue and Iceberg REST

Bugs and Testing

corleyma commented 1 week ago

@kevinzwang I am a bit confused about what copy-on-write writing represents in this set of features? Typically (at least coming from a Spark lens), copy-on-write is the default behavior for Iceberg v2 tables. Given that Daft supports basic and partitioned rights, what does it mean to say Daft does not have support for either COW or MOR writing? It has to be one or the other, right?

kevinzwang commented 1 week ago

@kevinzwang I am a bit confused about what copy-on-write writing represents in this set of features? Typically (at least coming from a Spark lens), copy-on-write is the default behavior for Iceberg v2 tables. Given that Daft supports basic and partitioned rights, what does it mean to say Daft does not have support for either COW or MOR writing? It has to be one or the other, right?

Hi @corleyma! COW and MOR are two different strategies we would need to support in order to do upsert (merge) operations in Iceberg. We currently only support append or overwrite for writing to Iceberg tables, which do not need either.

I've updated the roadmap to better reflect this.