CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Feature/experimental module #67

Closed vavison closed 5 years ago

vavison commented 5 years ago

Description

Introduces a new module, waimak-experimental, containing experimental features for which the APIs may change.

Adds to this new module functionality for running multiple Spark applications within a single main method whilst obeying configured dependency constraints. As part of this, two traits, SparkApp and its child WaimakApp provide a framework for writing Spark/Waimak applications in which the user need only implement the application logic and define the application's environment and configuration, with everything else taken care of.

Fixes #52

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Unit tests for all new functionality

coveralls commented 5 years ago

Pull Request Test Coverage Report for Build 490


Changes Missing Coverage Covered Lines Changed/Added Lines %
waimak-experimental/src/main/scala/com/coxautodata/waimak/spark/app/EnvironmentManager.scala 6 8 75.0%
waimak-experimental/src/main/scala/com/coxautodata/waimak/spark/app/MultiAppRunner.scala 15 17 88.24%
waimak-experimental/src/main/scala/com/coxautodata/waimak/spark/app/Env.scala 23 27 85.19%
<!-- Total: 59 67 88.06% -->
Totals Coverage Status
Change from base Build 484: 0.3%
Covered Lines: 1244
Relevant Lines: 1554

💛 - Coveralls