Naive-sql-engine not support spark-3.0.1 when running TPC-DS.

haojinIntel commented 3 years ago

When we using OAP-1.1 with spark-3.0.1 to run TPC-DS all queries, 19 queries failed. The error message is showed below:

Error: Error running query: java.lang.NoSuchMethodError: org.apache.spark.sql.execution.adaptive.ShuffleQueryStageExec.shuffle()Lorg/apache/spark/sql/execution/exchange/ShuffleExchangeLike; (state=,code=0)

haojinIntel commented 3 years ago

@zhixingheyi-tian @zhouyuan Please help to track the issue. Thanks!

zhouyuan commented 3 years ago

currently native sql does not have a API layer to connect with different spark version, there are two branches:

https://github.com/oap-project/native-sql-engine/tree/branch-1.1-spark-3.x -> 1.1 release supports spark 3.0.0 only
https://github.com/oap-project/native-sql-engine/tree/branch-1.1-spark-3.1.1 -> 1.1.1 release supports spark 3.1.1 only may need to create another branch to support Spark 3.0.1 if there's strong requirement

for future release we are considering to adding a compatible API layer https://github.com/oap-project/native-sql-engine/issues/273

oap-project / gazelle_plugin

Naive-sql-engine not support spark-3.0.1 when running TPC-DS. #323