search

qubole / spark-acid

ACID Data Source for Apache Spark based on Hive ACID
Apache License 2.0
97 stars 34 forks source link

sparkSQL Error : instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder #71

Open yukiHan0517 opened 4 years ago

yukiHan0517 commented 4 years ago

when I use assembly-0.5.0.jar to access acid table , it seems that dataframe API works well, reading or writing table , either partitioned or not.

But I can not use spark.sql() to access acid table，I init sparkSession following instruction :

val spark = SparkSession.builder() .appName("Hive-acid-test") .config("spark.sql.extensions", "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension") .enableHiveSupport() .<OTHER OPTIONS> .getOrCreate() it will not work above, But if I add the config after sparkSession build spark.conf.set("spark.sql.extensions", "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension") it seems used hiveacid.sql.execution.SparkAcidSqlParser to parse sql

but then an execption has been thrown: 20/06/23 22:12:18 ERROR ApplicationMaster: User class threw exception: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder': java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionStateBuilder': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1107) at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:145) at org.apache.spark.sql.SparkSession$$anonfun$sessionState$2.apply(SparkSession.scala:144) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:144) at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:141) at org.apache.spark.sql.SparkSession.conf$lzycompute(SparkSession.scala:170) at org.apache.spark.sql.SparkSession.conf(SparkSession.scala:170) at com.ccb.camp.load.SparkAcidTestSQL$.testReadSQL(SparkAcidTestSQL.scala:41) at com.ccb.camp.load.SparkAcidTestSQL$.main(SparkAcidTestSQL.scala:25) at com.ccb.camp.load.SparkAcidTestSQL.main(SparkAcidTestSQL.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:684) Caused by: java.lang.IllegalArgumentException: Can not set final org.apache.spark.sql.internal.VariableSubstitution field org.apache.spark.sql.execution.SparkSqlParser.substitutor to com.qubole.spark.datasources.hiveacid.sql.execution.SparkAcidSqlParser at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:167) at sun.reflect.UnsafeFieldAccessorImpl.throwSetIllegalArgumentException(UnsafeFieldAccessorImpl.java:171) at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:58) at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38) at java.lang.reflect.Field.get(Field.java:393) at com.qubole.spark.datasources.hiveacid.sql.execution.SparkAcidSqlParser.<init>(SparkAcidSqlParser.scala:35) at com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension$$anonfun$apply$3.apply(HiveAcidAutoConvert.scala:73) at com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension$$anonfun$apply$3.apply(HiveAcidAutoConvert.scala:72) at org.apache.spark.sql.SparkSessionExtensions$$anonfun$buildParser$1.apply(SparkSessionExtensions.scala:161) at org.apache.spark.sql.SparkSessionExtensions$$anonfun$buildParser$1.apply(SparkSessionExtensions.scala:160) at scala.collection.IndexedSeqOptimized$class.foldl(IndexedSeqOptimized.scala:57) at scala.collection.IndexedSeqOptimized$class.foldLeft(IndexedSeqOptimized.scala:66) at scala.collection.mutable.ArrayBuffer.foldLeft(ArrayBuffer.scala:48) at org.apache.spark.sql.SparkSessionExtensions.buildParser(SparkSessionExtensions.scala:160) at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser$lzycompute(BaseSessionStateBuilder.scala:117) at org.apache.spark.sql.internal.BaseSessionStateBuilder.sqlParser(BaseSessionStateBuilder.scala:116) at org.apache.spark.sql.internal.BaseSessionStateBuilder.build(BaseSessionStateBuilder.scala:292) at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$instantiateSessionState(SparkSession.scala:1104) ... 15 more 20/06/23 22:12:18 INFO SparkContext: Invoking stop() from shutdown hook

amoghmargoor commented 4 years ago

once sparkSession is created it should not allow to add spark.sql.extensions. I will fail like below: scala> spark.conf.set("spark.sql.extensions", "com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension") org.apache.spark.sql.AnalysisException: Cannot modify the value of a static config: spark.sql.extensions;

Which SparkVersion are you using and did you build jar from current master ? Moreover can you also try setting it via spark-shell arguments like this --conf spark.sql.extensions=com.qubole.spark.hiveacid.HiveAcidAutoConvertExtension and see if it works ?

amoghmargoor commented 4 years ago

hey @yukiHan0517 ... were you able to resolve the issue ?