search

tenzir / public-roadmap

The public roadmap of Tenzir
https://docs.tenzir.com/roadmap
4 stars 0 forks source link

Delta Table & Delta Lake Support #74

Open mavam opened 1 year ago

mavam commented 1 year ago

This roadmap item is about Delta Table support.

## Definition of Done
- [ ] Investigate options to [convert Parquet files to Delta Tables](https://docs.delta.io/latest/delta-utility.html#convert-a-parquet-table-to-a-delta-table)
- [ ] Implement a `delta_table` printer that writes Delta Tables
- [ ] Figure out how to "commit" Delta Tables to Delta Lake and identify a pipeline-native way to do so reliably

Regarding the last deliverable: as we get to know the API of Delta Lake, we should figure out whether our connector-format API is sufficient here. For example, it would be nice if we could write to delta-lake write delta-table, or even to delta-lake write parquet (since we already have Parquet).

mavam commented 1 year ago

I got this in a DM:

Delta lake support is a magic keyword for us as that's one of the main reason we went with Databricks