CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Change force recreate tables flag to waimak configuration parameter #68

Closed alexjbush closed 5 years ago

alexjbush commented 5 years ago

Description

This PR moves the force recreate tables flag in the HadoopDBConnector trait to a Waimak configuration option.

This PR also introduces the foldLeftOver on the DataFlow trait to make it easier to fold over a collection type using a DataFlow as a zero-value.

Fixes #65

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Unit tests

coveralls commented 5 years ago

Pull Request Test Coverage Report for Build 480


Changes Missing Coverage Covered Lines Changed/Added Lines %
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlow.scala 0 1 0.0%
waimak-impala/src/main/scala/com/coxautodata/waimak/metastore/ImpalaDBConnector.scala 1 4 25.0%
waimak-core/src/main/scala/com/coxautodata/waimak/metastore/MetastoreUtils.scala 0 4 0.0%
<!-- Total: 13 21 61.9% -->
Files with Coverage Reduction New Missed Lines %
waimak-rdbm-ingestion/src/main/scala/com/coxautodata/waimak/rdbm/ingestion/SQLServerTemporalExtractor.scala 1 83.33%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/StorageActions.scala 1 93.75%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/FileStorageOps.scala 1 92.31%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/ParallelActionScheduler.scala 1 96.55%
waimak-rdbm-ingestion/src/main/scala/com/coxautodata/waimak/rdbm/ingestion/RDBMIngestionActions.scala 1 96.15%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlow.scala 2 82.26%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala 2 95.32%
waimak-core/src/main/scala/com/coxautodata/waimak/log/Logging.scala 2 23.08%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlow.scala 2 93.33%
<!-- Total: 13 -->
Totals Coverage Status
Change from base Build 461: -0.5%
Covered Lines: 1181
Relevant Lines: 1485

💛 - Coveralls