search

delta-io / delta

An open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs
https://delta.io
Apache License 2.0
7.62k stars 1.71k forks source link

[Spark] Replace the default pattern matching for LogicalRelation to LogicalRelationWithTable #3805

Closed HeartSaVioR closed 3 weeks ago

HeartSaVioR commented 3 weeks ago

Which Delta project/connector is this regarding?

Description

This PR proposes to replace the default pattern matching for LogicalRelation to newly introduced pattern object LogicalRelationWithTable which will be available in upcoming Spark 4.0.

This change helps the project to modify less pieces of code when Spark makes the change to the LogicalRelation; most pattern matchings with LogicalRelation only extract the relation and catalog table, hence they fit with LogicalRelationWithTable.

How was this patch tested?

Existing tests would suffice.

Does this PR introduce any user-facing changes?

No.

HeartSaVioR commented 3 weeks ago

I'm planning to propose the change of LogicalRelation (new parameter of constructor) in Spark 4.0, which will break all pattern matches. For creating an instance of LogicalRelation, I'll try to figure out the way to be compatible with existing constructor (and be back here to address the issue with shim if I fail). But for pattern matching, every addition of parameter require patterns to be updated. It won't be needed after this PR.