Open jalpan-randeri opened 9 months ago
@jalpan-randeri Do we plan to merge this, this can fix following issue also https://github.com/snowflakedb/spark-snowflake/issues/567
@sfc-gh-bli Do we plan to merge this PR?
Yes, i plan to merge this. However I am waiting for review of this PR. Can you review it?
Yes, i plan to merge this. However I am waiting for review of this PR. Can you review it?
Overall, it looks good, but I am not commiter, so you need approval from someone from snow
@sfc-gh-bli please review and share your thoughts
Under Adaptive query execution mode, Spark overlaps planning and execution phase, This results in spark running planning multiple time. The current implementation eagerly pushdown query in planning stage, this result into redundant query pushdown to snowflake and it ignores the runtime discovered filters.
This commit handles this scenario and delayed pushdown, This gives Spark AQE a chance to generate the most optimum plan and eliminating the pushdown of redundant queries. This results in improved performance as new filters identified at runtime by AQE are pushdown.
Furthermore, it logs the pushdown query into spark plan. This allow easy debugging from Spark History Server and UIs and from logs.
This PR adds new unit test suit for it.