search

hortonworks-spark / spark-llap

Apache License 2.0
102 stars 68 forks source link

Refactor configuration handling on branch-2.3-3.0 #227

Closed EricWohlstadter closed 6 years ago

EricWohlstadter commented 6 years ago

Refactor to conform with org.apache.spark.sql.sources.v2.SessionConfigSupport. Previously this only worked for DataSourceReader, because usage of DataFrameReader was hidden behind HiveWarehouseSession.

While testing with DataSourceWriter, I see the current structure won't work well without a lot of hacks.

Refactoring to cleanup and make configuration more general and predictable.

There are two ways for configuration to be specified:

  1. In static config (e.g. spark-defaults), keys with the prefix "spark.datasources.hive.warehouse" will get picked up automagically and moved into the SQL session config.
  2. In HiveWarehouseBuilder, options can be specified. These will override anything in the static conf.