search

databricks / spark-sql-perf

Apache License 2.0
586 stars 407 forks source link

Not support partitionTables option #26

Closed libratiger closed 9 years ago

libratiger commented 9 years ago

I used this * tables.genData(location, format, true, true, true, true, true)* and occurred the following problem, it is ok for tables.genData(location, format, true, false, true, true, true), so this benchmark do not support partitionTables?

java.lang.RuntimeException: [7.1] failure: ``union'' expected but identifier DISTRIBUTE found

After I read the source code, I found the problem occurred on this line. val query = s""" |SELECT | $columnString |FROM | $tempTableName |$predicates |DISTRIBUTE BY | $partitionColumnString """.stripMargin

Is there any suggestion to solve this problem?

I run this on Spark1.5.

libratiger commented 9 years ago

I am not familiar with Spark SQL, In my experiment, it seem that Spark SQL can not support DROP command.

If I use the tpcds.createResultsTable(), there will a error occurred similar to the above.

marmbrus commented 9 years ago

you need to use a HiveContext.

libratiger commented 9 years ago

Thanks! Because I do not find the difference in the document at first.