search

apache / incubator-gluten

Gluten is a middle layer responsible for offloading JVM-based SQL engines' execution to native engines.
https://gluten.apache.org/
Apache License 2.0
1.22k stars 438 forks source link

[GLUTEN-7812][CH] Fix the query failed for the mergetree format when the 'spark.databricks.delta.stats.skipping' is off #7813

Closed zzcclp closed 2 weeks ago

zzcclp commented 3 weeks ago

What changes were proposed in this pull request?

For the Spark 3.3 + Delta 2.3, when the 'spark.databricks.delta.stats.skipping' is off, there are some queries failed with the subquery, the error message is below:

java.lang.IllegalArgumentException: requirement failed
    at scala.Predef$.require(Predef.scala:268)
    at org.apache.spark.sql.delta.SubqueryTransformerHelper.transformWithSubqueries(SubqueryTransformerHelper.scala:42)
    at org.apache.spark.sql.delta.SubqueryTransformerHelper.transformWithSubqueries$(SubqueryTransformerHelper.scala:40)
    at org.apache.spark.sql.delta.stats.PrepareDeltaScan.transformWithSubqueries(PrepareDeltaScan.scala:291)
    at org.apache.spark.sql.delta.PreprocessTableWithDVs.preprocessTablesWithDVs(PreprocessTableWithDVs.scala:67)
    at org.apache.spark.sql.delta.PreprocessTableWithDVs.preprocessTablesWithDVs$(PreprocessTableWithDVs.scala:66)
    at org.apache.spark.sql.delta.stats.PrepareDeltaScan.preprocessTablesWithDVs(PrepareDeltaScan.scala:291)
    at org.apache.spark.sql.delta.stats.PrepareDeltaScanBase.apply(PrepareDeltaScan.scala:227)
    at org.apache.spark.sql.delta.stats.PrepareDeltaScanBase.apply$(PrepareDeltaScan.scala:191)
    at org.apache.spark.sql.delta.stats.PrepareDeltaScan.apply(PrepareDeltaScan.scala:291)
    at org.apache.spark.sql.delta.stats.PrepareDeltaScan.apply(PrepareDeltaScan.scala:291)

Close #7812.

(Fixes: #7812)

How was this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)

(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)

github-actions[bot] commented 3 weeks ago

https://github.com/apache/incubator-gluten/issues/7812

github-actions[bot] commented 3 weeks ago

Run Gluten Clickhouse CI

zzcclp commented 3 weeks ago

Run Gluten Clickhouse CI

github-actions[bot] commented 2 weeks ago

Run Gluten Clickhouse CI on x86

github-actions[bot] commented 2 weeks ago

Run Gluten Clickhouse CI on x86

zzcclp commented 2 weeks ago

Run Gluten Clickhouse CI on x86