search

yaooqinn / spark-authorizer

A Spark SQL extension which provides SQL Standard Authorization for Apache Spark | This repo is contributed to Apache Kyuubi | 项目已迁移至 Apache Kyuubi
https://yaooqinn.github.io/spark-authorizer/
Apache License 2.0
171 stars 79 forks source link

Spark Thrift Server cannot use external metastore #18

Open kk17 opened 5 years ago

kk17 commented 5 years ago

Describe the bug I am running standalone spark version 2.3.2 in docker with ranger-hive-plugin 0.5.3. I use a extend MySQL database for store hive metastore data. I found out that Spark Thrift Server cannot use external metastore for excuting SQL statements after enabling spark configuration: spark.sql.extensions=org.apache.ranger.authorization.spark.authorizer.RangerSparkSQLExtension . If I remove the above configuration Spark Thrift Server can use external metastore but the ranger-hive-plugin is not enabled. This issue does not happen when using spark-sql and spark-shell.

To Reproduce Steps to reproduce the behavior: copied ranger.xml conf files to spark_home/conf, copied ranger-hive.jars to spark_home/jars along with spark-authorizer jar. gave full permissions to all xml and jar files. Modified conf xml files as advised. Environments are spark standalone 2.3.2, with ranger-hive-plugin 0.5.3, ranger-admin 0.5.3. Tried running show databases command inbeeline.

Expected behavior Spark Thrift Server can use external metastore after enable ranger-hive-plugin.

Additional context Gist for configuration files: Spark-SQL-Ranger

yaooqinn commented 5 years ago

Spark Thrift Server is not guaranteed you may try kyuubi https://github.com/yaooqinn/kyuubi