CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Feature/generic properties providers #62

Closed alexjbush closed 5 years ago

alexjbush commented 5 years ago

Description

This PR provides a generic way to provide configuration implementations. User can now provide a list of objects to use when looking for configuration options in the case class config parser. Provided implementations are using a properties file stored on a Hadoop-compatible FS and Databricks secrets store.

This change is breaking to the parser API, as it now needs a flow context.

The Databricks implementation is in a new module, and only compiled for Scala 2.11 as there is no 2.12 library available.

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Unit tests and integration test in Databricks Notebook.

coveralls commented 5 years ago

Pull Request Test Coverage Report for Build 455


Changes Missing Coverage Covered Lines Changed/Added Lines %
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/FlowContext.scala 0 1 0.0%
<!-- Total: 40 41 97.56% -->
Files with Coverage Reduction New Missed Lines %
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/StorageActions.scala 1 93.75%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkInterceptors.scala 1 94.44%
waimak-hive/src/main/scala/com/coxautodata/waimak/metastore/HiveDBConnector.scala 1 96.43%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/FileStorageOps.scala 1 90.77%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala 1 95.91%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/ParquetDataCommitter.scala 1 96.67%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkActions.scala 2 70.37%
waimak-core/src/main/scala/com/coxautodata/waimak/log/Logging.scala 2 30.77%
<!-- Total: 10 -->
Totals Coverage Status
Change from base Build 442: 0.6%
Covered Lines: 1190
Relevant Lines: 1485

💛 - Coveralls