search

databricks / spark-sql-perf

Apache License 2.0
586 stars 407 forks source link

NoSuchMethodError on Spark 3.1 in Databricks #202

Open mithun1979 opened 3 years ago

mithun1979 commented 3 years ago

Hi, We are trying to run TPC-DS queries on spark-sql-perf on Spark 3.1. We tried in Databricks runtimes 7.3 LTS and 8.1. We are using the JAR - spark-sql-perf_2.12-0.5.1-SNAPSHOT.jar We are getting the following error:

java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.UnresolvedRelation.tableIdentifier()Lorg/apache/spark/sql/catalyst/TableIdentifier; logs.txt

Please find the detailed logs attached. Would you be able to help with guidance on this issue.

Thanks and Regards Saurav Chakraborty

mithun1979 commented 3 years ago

Just wanted to mention that we are running the following code. Its runs fine of Databricks runtion 5.5 LTS (Spark 2.4) with the JAR spark-sql-perf_2.11-0.5.1-SNAPSHOT.jar.

import com.databricks.spark.sql.perf.tpcds.TPCDS

val tpcds = new TPCDS (sqlContext = sqlContext) def queries = { val filtered_queries = query_filter match { case Seq() => tpcds.tpcds24Queries case => tpcds.tpcds2_4Queries.filter(q => query_filter.contains(q.name)) } if (randomizeQueries) scala.util.Random.shuffle(filtered_queries) else filtered_queries }

val experiment = tpcds.runExperiment( queries, iterations = iterations, resultLocation = resultLocation, forkThread = true)

println(experiment.toString) experiment.waitForFinish(timeout6060)