CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Issues #53 Allow repartition by integer value #57

Closed alexjbush closed 5 years ago

alexjbush commented 5 years ago

Description

This change introduces the option of staging a commit with a numeric repartition value.

Also introduces a get conf mechanism on the flow context, and moves the cache commit option to a global conf.

Fixes #53

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Unit tests

Note

The wiki pages for commit actions and parameters will need to be updated

coveralls commented 5 years ago

Pull Request Test Coverage Report for Build 405


Changes Missing Coverage Covered Lines Changed/Added Lines %
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/FlowContext.scala 2 5 40.0%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkActions.scala 6 9 66.67%
<!-- Total: 34 40 85.0% -->
Files with Coverage Reduction New Missed Lines %
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlow.scala 2 82.26%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkActions.scala 2 70.37%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlow.scala 3 94.03%
<!-- Total: 7 -->
Totals Coverage Status
Change from base Build 400: 0.5%
Covered Lines: 1138
Relevant Lines: 1440

💛 - Coveralls