delta-io / delta-sharing

An open protocol for secure data sharing
https://delta.io/sharing
Apache License 2.0
774 stars 173 forks source link

Introduce pattern object for LogicalRelation to avoid coupling with all params #603

Closed HeartSaVioR closed 2 weeks ago

HeartSaVioR commented 2 weeks ago

DeltaSharingLimitPushDown uses the pattern with LogicalRelation which could change, in terms of parameter in constructor. Spark community made a change in upcoming Spark 4.0 (https://github.com/apache/spark/pull/48676) to add a new parameter in LogicalRelation, and the change will break the pattern.

Since the pattern only picks up the relation from LogicalRelation, the pattern object Spark community introduced along with the above change (https://github.com/apache/spark/pull/48605) would be helpful to not encounter the issue in upcoming Spark 4.0 when Delta Sharing supports the version.