[Catalogs] [Delta Lake] Add support for updating an existing Delta Lake table (overwrite, append, upsert, delete).

[clarkzinzow]

In addition to creating new Delta Lake tables, we should add support for updating existing Delta Lake tables. We should support overwrites, appends, upserts, and deletes.

• https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.update
• https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.merge
• https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.delete
• https://docs.delta.io/latest/delta-update.html#table-deletes-updates-and-merges
• https://github.com/delta-io/delta/blob/master/PROTOCOL.md

[sujiplr]

Are we planning this supports sometimes next 3 months?

[samster25]

@sujiplr yes we are, once we do our integration with delta-kernel-rs.

cc: @kevinzwang

[kevinzwang]

Hi @sujiplr! Glad to hear that there's demand for these features. We currently have support for overwrite and append with the DataFrame.write_deltalake method, and upserts and deletes are part of our delta lake roadmap (#2457) and we hope to support them soon

[kevinzwang]

In the meantime, if your data can fit on one machine, you can call DataFrame.to_arrow to convert your dataframe into a PyArrow table, and then use it with the deltalake python library

[kevinzwang]

I think I'll also close this issue since each of these features are either done or being tracked by another issue already