search

viaduct-ai / docker-spark-k8s-aws

Docker image for running Spark 3 on Kubernetes on AWS
27 stars 14 forks source link

No access to glue #2

Open parisni opened 3 years ago

parisni commented 3 years ago

is this supposed to access to aws glue catalog ? I get those warning together with an empty local metastore.

I made working spark 2.4.x + hive 1.2.1. Sadly no success with spark3 and your work. 

scala> spark.sql("show databases").show
21/08/09 15:44:47 WARN HiveConf: HiveConf of name hive.metastore.client.factory.class does not exist
21/08/09 15:44:48 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/08/09 15:44:48 WARN HiveConf: HiveConf of name hive.metastore.client.factory.class does not exist
21/08/09 15:44:48 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/08/09 15:44:49 WARN HiveConf: HiveConf of name hive.metastore.client.factory.class does not exist
21/08/09 15:44:53 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/08/09 15:44:53 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment
+---------+
|namespace|
+---------+
|  default|
+---------+
parisni commented 3 years ago

I finally got this working by using this https://github.com/kuhnen/spark-glue/tree/to_docker