CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Feature/commit and cleanup #34

Closed alexeipab closed 5 years ago

alexeipab commented 5 years ago

Description

Adds flexible commit definitions that a separate from the push implementations. Enables multiple commit definitions. It also cleans up generics inheritance and error reporting of the data flow validation phases.

Fixes # (issue)

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Unit tested.

coveralls commented 5 years ago

Pull Request Test Coverage Report for Build 280


Changes Missing Coverage Covered Lines Changed/Added Lines %
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlowExecutor.scala 6 7 85.71%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/InterceptorAction.scala 0 1 0.0%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlow.scala 10 11 90.91%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlowAction.scala 1 2 50.0%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/EmptyFlowContext.scala 1 3 33.33%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/ParallelDataFlowExecutor.scala 2 4 50.0%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/ParquetDataCommitter.scala 47 49 95.92%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlow.scala 70 74 94.59%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DFExecutorPriorityStrategies.scala 13 18 72.22%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SimpleSparkDataFlow.scala 18 23 78.26%
<!-- Total: 195 219 89.04% -->
Files with Coverage Reduction New Missed Lines %
waimak-core/src/main/scala/com/coxautodata/waimak/log/Logging.scala 1 30.77%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkActions.scala 1 69.15%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlow.scala 2 84.78%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlow.scala 2 95.28%
<!-- Total: 6 -->
Totals Coverage Status
Change from base Build 268: 1.3%
Covered Lines: 1013
Relevant Lines: 1298

💛 - Coveralls