Roadmap 2022 H2 (discussion)

dennyglee commented 2 years ago

This is a working issue for folks to provide feedback on the prioritization of the Delta Lake priorities spanning July to December 2022. With the release of Delta Lake 2.0, we wanted to take the opportunity to discuss other vital features for prioritization with the community based on the feedback from the Delta Users Slack, Google Groups, Community AMAs (on Delta Lake YouTube), the Roadmap 2022H2 (discussion), and more.

Note, tasks that are crossed out (i.e., ~00~) have been completed.

To review the Delta Rust roadmap only, please refer to https://go.delta.io/rust-roadmap for more information.

Priority 0

We will focus on these issues and continue to deliver parts (or all of the issue) over the next six months

Issue	Category	Task	Description	Status
~256~	Flink	Flink Source	Build Flink source to read Delta tables in batch and streaming jobs	In Progress
238	Flink	Flink SQL+ Table API + Catalog Support	After Flink Sink and Source, build support for Flink Catalog, SQL, and Table API	In Progress
411, 410	Flink	Productionize support for all cloud object stores	Make sure that Flink Sink can write robustly to S3, GCS, ADLS2 with full transactional guarantees	In Progress
~610~	Rust	Integrate with a common object-store abstraction from arrow / Rust ecosystem	This will allow us to provide a more convenient and performant API on the Rust and python side	In Progress
~575~	Rust	Support V2 writer protocol	Utilize PyArrow-based writer function (write_deltalake) support writer protocol V2 and object stores S3, GCS, and ADLS2.	In Progress
~761~	Rust	Expand write support for cloud object stores	Write to object stores S3, GCS, and ADLS2 from multiple clusters with full transactional guarantees	In Progress
	Rust	DAT Integration	Delta Acceptance Tests running in CI	In Progress
	Rust	Rust documentation	First pass at Rust docs	In Progress
	Rust	Rust blogging	Blog post for the Rust community	In Progress
632	Rust	Commit protocol	Fully protocol compliant optimistic commit protocol	In Progress
851	Rust	Rust writer	Refactor Rust writer API to be flexible for others wishing to build upon delta-rs	In Progress
~1257~	Spark	Release Delta 2.1 on Apache Spark 3.3	Ensure the latest version of Delta Lake works with the latest version of Apache Spark™	Released in Delta 2.1
~1485~	Spark	Support reading tables with Deletion Vectors	Allow reads on tables that have deletion vectors to mark rows in parquet files as removed.	Released in Delta 2.3
~1408~	Spark	Support Table Features protocol	Upgrade the protocol to use Table Features to specify the features needed to read/write to a table.	Released in Delta 2.3
~1242~	Spark	Support time travel SQL syntax	Delta currently supports time travel via Python and Scala APIs. We would like to extend support for the SQL syntax `VERSION AS OF` and `TIMESTAMP AS OF` in SELECT statements.	Released in Delta 2.1
	Standalone	Extend Delta Standalone for higher protocol versions	Extend Delta Standalone to support logs using higher protocol versions and advanced features like constraints, generated columns, column mapping, etc.	In Progress
	Standalone	Expand support for data skipping in Delta Standalone	Extend the current data skipping to skip file using column stats and more expressions	In Progress
	Website	Updated Delta Lake documentation	Move Delta Lake documentation to the website GitHub repo to allow easier community collaboration	In Progress
	Website	Consolidate all connector documentation	Consolidate docs of all connectors in the website Github repo	In Progress

Priority 1

We should be able to deliver parts (or all of the issue) over the next six months

Issue	Category	Task	Description	Status
4	Core	Delta Acceptance Testing (DAT)	With various languages interacting with the Delta protocol (e.g., Delta Standalone, Delta Spark, Delta Rust, Trino, etc.), we propose to have the same reference tables and library of reference tests to ensure all Delta APIs remain in compliance.	In Progress
1347	Core	Support Bloom filters	Improve query performance by utilizing bloom filters. The approach is TBD due to recent updates to Apache Parquet to support bloom filters.	Not Started
1387	Core	Enable Delta clone	Clones a source Delta table to a target destination at a specific version. A clone can be either deep or shallow: deep clones copy over the data from the source and shallow clones do not.	Shallow clone is released in Delta 2.3
	Delta connectors	GoLang Delta connector	Support GoLang reading a Delta Lake table natively	Not Started
	Delta connectors	Improve partition filtering in Power BI client	Improved partition filtering using built-in UI filters in Power BI	Not Started
	Delta connectors	Pulsar Source connector	Support Apache Pulsar reading a Delta Lake table natively	Not Started
	Flink	Column stats generation in Flink Sink	Make the Flink Delta sink generate column stats	Not Started
	Presto/Trino	Support higher protocol versions in Presto and Trino	Use Standalone to support higher protocol versions	Not Started
	Rust	Delta Rust API Updates	Update APIs and support more high-level operations on top of delta; this includes better conflict resolution	NA
	Rust	Better support for large logs	Better support for handling large Delta logs/snapshots	NA
	Sharing Connectors	GoLang Delta Sharing client	Support GoLang client for Delta Sharing	NA
	Sharing Connectors	R Delta Sharing client	Support R client for Delta Sharing	NA
1072	Spark	Support for Identity columns	Create an identity column that will be automatically assigned a unique and statistically increasing (or decreasing if the step is negative) value.	Not Started
	Spark	Support querying Change Data Feed (CDF) using SQL queries	To support querying CDF using SQL queries in Apache Spark, we need to allow custom TVFs to be resolved using injected rules.	Released in Delta 2.3
1156	Spark	Support Auto Compaction	Provide auto compaction functionality to simplify compaction tasks	In Progress
1198	Spark	Support Optimize Writes	Optimize Spark to Delta Lake writes	In Progress
~1462~	Spark	Enable converting from Iceberg to Delta	Enable converting parquet-backed Iceberg tables to Delta tables without rewriting parquet files.	Released in Delta 2.3
~1464~	Spark	Shallow clone Iceberg tables	Enable shallow cloning parquet-backed Iceberg tables following the Delta protocols without the need to copy all of the data.	Released in Delta 2.3
~1349~	Spark	Improve semantics of column mapping and Change Data Feed	Improve semantics of how column renames/drops (aka column mapping) interact with CDF and streaming	Released in Delta 2.3

Priority 2

Issue	Category	Task	Description
Sharing	Share individual partitions	Support Sharing individual partitions in Delta Sharing	NA
Sharing Connectors	Rust Delta Sharing client	Support Rust client for Delta Sharing	NA
Sharing Connectors	Starburst/Trino Delta Sharing connector	Support Starburst/Trino client for Delta Sharing	NA
Sharing Connectors	Airflow Delta Sharing connector	Support sharing data from Airflow sensor	NA
Rust	Process	Release improvements	NA

History

2022-08-01: Initial creation
2022-08-02: Delta Sharing updates
2022-08-08: Include Identity columns in the roadmap
2022-09-13: Update issues and include into roadmap auto compaction, optimize writes, and bloom filters.
2022-09-19: Update to include Delta Clone
2022-09-22: Including working Delta Rust roadmap document
2022-10-26: Included updated Delta Rust roadmap in GitHub link
2022-10-27: Included converting and shallow cloning Iceberg to Delta

dennyglee commented 2 years ago

Note, we will be adding/updating the issue over the next few weeks but I'm a little behind schedule so thought I would get the roadmap discussion started ASAP. Thanks!

edfreeman commented 2 years ago

Hi folks. Can't see it explicitly mentioned so thought I'd ask - will identity columns support (i.e. writer version 6) be added in this H2 wave? That's a big feature we're keen to be able to use outside of Databricks, and it didn't quite make it into 2.0 by the looks of things.

dennyglee commented 2 years ago

Hi folks. Can't see it explicitly mentioned so thought I'd ask - will identity columns support (i.e. writer version 6) be added in this H2 wave? That's a big feature we're keen to be able to use outside of Databricks, and it didn't quite make it into 2.0 by the looks of things.

Thanks for the call out @edfreeman - identity columns has been added :)

sezruby commented 2 years ago

Hi @dennyglee, what about Auto compaction and Optimize Write? I don't think the PRs are getting some attention for review / merge. Could you add it to the roadmap?

dennyglee commented 2 years ago

Oh good point! Let me get back to you on this shortly! Sorry about that!

On Wed, Aug 17, 2022 at 18:41 EJ Song @.***> wrote:

Hi @dennyglee https://github.com/dennyglee, what about Auto compaction and Optimize Write? I don't think the PRs are getting some attention for review / merge. Could you add it to the roadmap?

— Reply to this email directly, view it on GitHub https://github.com/delta-io/delta/issues/1307#issuecomment-1218895025, or unsubscribe https://github.com/notifications/unsubscribe-auth/AALBHLLDOCHFYXAGKWGKXEDVZWIC3ANCNFSM55JXKP7A . You are receiving this because you were mentioned.Message ID: @.***>

keen85 commented 2 years ago

Hi @dennyglee, what about support for displaying DDL of delta tables (SHOW CREATE TABLE) https://github.com/delta-io/delta/issues/1032 https://github.com/delta-io/delta/pull/1255

dennyglee commented 2 years ago

Hi @dennyglee, what about support for displaying DDL of delta tables (SHOW CREATE TABLE) #1032 #1255

Good call out @keen85 - let me check with @zpappa on this!

zpappa commented 2 years ago

Hi @dennyglee, what about support for displaying DDL of delta tables (SHOW CREATE TABLE) #1032 #1255

Good call out @keen85 - let me check with @zpappa on this!

I have some minor style and test updates for this PR to be considered done, I can finish them today and we can try to pull them Into 2.1

dudzicp commented 2 years ago

How about delta caching that is present on databricks? Is there a plan for such feature?

tdas commented 2 years ago

"Delta caching" is actually a Databricks Runtime engine feature, not part of the format. Caching data on an processing engine's executor/workers nodes is something that can really be done well by the engine itself, not by a data format. It's unfortunate and confusing that we had marketed it under the "Delta" brand name, even though it's really not part of the "Delta Lake" storage format. So, in short, its not really possible to open source that as part of Delta Lake.

djouallah commented 2 years ago

I Have being experimenting with Delta lake in Google Cloud and DuckDB and it is very promising, but without a local SSD cache it will never be fast enough, maybe we need a cache for Delta independently from Databricks implementation, Delta knows which files needs to be scanned, keeping a local copy on the first call will be really useful, at least for a the standalone reader

khwj commented 2 years ago

Hi @dennyglee, what about supporting analyze table #581 and Bloom filter indexes#1347?

MaksGS09 commented 2 years ago

Hi! How about Big Query integration?

sezruby commented 2 years ago

Hi @dennyglee any update?

dennyglee commented 2 years ago

Sorry about that @sezruby - yes, we will be adding these to the roadmap very shortly. Thanks for your patience (I’ve been out the last two weeks)

dennyglee commented 2 years ago

Some quick updates:

@sezruby Thanks for your patience - auto compaction and optimize writes have been included to the roadmap.
@khwj included bloom filters and added some comments directly to your issue
@MaksGS09 We're still determining the resources required for performant BQ integration
We are currently reviewing #1255 and #1377

HTH!

p2bauer commented 1 year ago

Hi @dennyglee ! I know it was on the 2022 H1 github page, but I haven't seen any mention on clone functionality being moved into the open source library. Is there any update around that? I poked around the current source code but didn't really see it anywhere.

dennyglee commented 1 year ago

Thanks @p2bauer - great call out. I've added this to the roadmap and created issue #1387 to track this. HTH!

SanthoshPrasad commented 1 year ago

Hi @dennyglee , Is there any update on supporting higher protocol versions in Presto and Trino?

dennyglee commented 1 year ago

Great question @SanthoshPrasad - we've been working with the PrestoDB and the Trino communities on this and we should have some updates on various progress around this over the next couple of months. One of the methods we're doing this is through our DAT effort (Delta Acceptance Testing) so we can more cleanly document and clarify which APIs are on which protocol version. If you're interested in learning more on this, please join us in the #dat channel in Delta Users Slack. HTH!

dennyglee commented 1 year ago

Suggest we add Airbyte Destination S3: add delta lake/delta table support to the roadmap as it's already part of the Delta Rust Roadmap - WDYT?

melin commented 1 year ago

Support jdbc catalog https://github.com/delta-io/delta/issues/1459

keen85 commented 1 year ago

I'd like to suggest adding "Register VACUUM in delta log" to the roadmap

dudzicp commented 1 year ago

I know that each commit, min/max values are calculated for each parquet file and are present in the delta log json, but how about adding more granularity to existing data skipping mechanism, by using parquet page skipping? Relevant links:

Would this be doable?

dennyglee commented 1 year ago

I know that each commit, min/max values are calculated for each parquet file and are present in the delta log json, but how about adding more granularity to existing data skipping mechanism, by using parquet page skipping? Relevant links:

https://issues.apache.org/jira/browse/PARQUET-922

https://blog.cloudera.com/speeding-up-select-queries-with-parquet-page-indexes/

Would this be doable?

@dudzicp Oh, could you please create a separate issue for this and we can discuss the specifics there? Thanks!

dudzicp commented 1 year ago

How about bucketing?

benbauer89 commented 1 year ago

Hey, I am interested in more details regarding https://delta.io/sharing/ It's stated that presto and trino are coming soon, but I could not really find any details or timelines. Please notice, that I am asking regaring delta sharing in context of Uniyt Catalog in particular and not necessarily regarding delta & trino/presto integration

dennyglee commented 1 year ago

Ahh, for Delta Sharing features within the context of UC, please ping Databricks community. Thanks!

robertkossendey commented 1 year ago

Hey @dennyglee any updates on the Roadmap? :) I was creating some issues in the mack project (e.g. Python support for table property update) but wanted to make sure that the delta-spark team is not working on the things that I came up with already.

dennyglee commented 1 year ago

Thanks for your patience @robertkossendey - we're working on this but admittedly way behind schedule due to all of the various asks, eh?! Saying this, please continue working on mack project activities as those are the ones we're pretty sure make more sense for mack to address or at least if we plan to merge this into delta-spark, it'll be further out on the roadmap. Thanks for the ping, eh?!

robertkossendey commented 1 year ago

Good to know, thank you for the update @dennyglee!

felipepessoto commented 1 year ago

@dennyglee @allisonport-db do you have any updates on auto compact and optimize write?

delta-io / delta