DeltaSharingLimitPushDown uses the pattern with LogicalRelation which could change, in terms of parameter in constructor. Spark community made a change in upcoming Spark 4.0 (https://github.com/apache/spark/pull/48676) to add a new parameter in LogicalRelation, and the change will break the pattern.
Since the pattern only picks up the relation from LogicalRelation, the pattern object Spark community introduced along with the above change (https://github.com/apache/spark/pull/48605) would be helpful to not encounter the issue in upcoming Spark 4.0 when Delta Sharing supports the version.
604
This PR introduces a shim to address the new parameter in LogicalRelation in Spark 4.0.
Since Delta Sharing does not perform cross compilation for Spark version and Spark 4.0 is expected to be released in early next year, this PR would help to defer the effort of cross compilation with multiple artifacts till really needed.
DeltaSharingLimitPushDown uses the pattern with LogicalRelation which could change, in terms of parameter in constructor. Spark community made a change in upcoming Spark 4.0 (https://github.com/apache/spark/pull/48676) to add a new parameter in LogicalRelation, and the change will break the pattern.
Since the pattern only picks up the relation from LogicalRelation, the pattern object Spark community introduced along with the above change (https://github.com/apache/spark/pull/48605) would be helpful to not encounter the issue in upcoming Spark 4.0 when Delta Sharing supports the version.
This PR introduces a shim to address the new parameter in LogicalRelation in Spark 4.0.
Since Delta Sharing does not perform cross compilation for Spark version and Spark 4.0 is expected to be released in early next year, this PR would help to defer the effort of cross compilation with multiple artifacts till really needed.