search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[Table Redirect] Introduce Redirect ReaderWriter Feature #3801

Closed kamcheungting-db closed 3 weeks ago

kamcheungting-db commented 4 weeks ago

Which Delta project/connector is this regarding?

Description

This PR introduces a new reader-writer table feature "redirection". This table feature would redirect the read and write query from the current storage location to a new storage location described inside the value of table feature.

The redirection has several phases to ensure no anomaly. To label these phases, we introduces four states:

  1. NO-REDIRECT: This state indicates that redirect is not enabled on the table.
  2. ENABLE-REDIRECT-IN-PROGRESS: This state indicates that the redirect process is still going on. No DML or DDL transaction can be committed to the table when the table is in this state.
  3. REDIRECT-READY: This state indicates that the redirect process is completed. All types of queries would be redirected to the table specified inside RedirectSpec object.
  4. DROP-REDIRECT-IN-PROGRESS: The table redirection is under withdrawal and the redirection property is going to be removed from the delta table. In this state, the delta client stops redirecting new queries to redirect destination tables, and only accepts read-only queries to access the redirect source table.

To ensure no undefined behavior, the valid procedures of state transition are:

  1. NO-REDIRECT -> ENABLE-REDIRECT-IN-PROGRESS
  2. ENABLE-REDIRECT-IN-PROGRESS -> REDIRECT-READY
  3. REDIRECT-READY -> DROP-REDIRECT-IN-PROGRESS
  4. DROP-REDIRECT-IN-PROGRESS -> NO-REDIRECT
  5. ENABLE-REDIRECT-IN-PROGRESS -> NO-REDIRECT

The protocol RFC document is on: https://github.com/delta-io/delta/issues/3702

How was this patch tested?

Unit Test of transition between different states of redirection.

Does this PR introduce any user-facing changes?

No

dennyglee commented 3 weeks ago

Hi @kamcheungting-db - could you check why the Delta Spark Master check if failing here?

vkorukanti commented 3 weeks ago

Hi @kamcheungting-db - could you check why the Delta Spark Master check if failing here?

Delta Spark Master compilation is broken for a while. Currently, it compiles against Spark master, which made some significant API changes that we need to accommodate in Delta.