search

CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Feature/refactor scheduler #99

Closed vavison closed 4 years ago

vavison commented 4 years ago

Description

Refactored executor and scheduler implementations to be more functional, to make it easier in future to look at making fail behaviour configurable.

How Has This Been Tested?

Existing unit tests are passing. Also tested on a complicated job which ran without issues.

codecov-io commented 4 years ago

Codecov Report

Merging #99 into develop will increase coverage by 0.09%. The diff coverage is 100%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop      #99      +/-   ##
===========================================
+ Coverage    88.32%   88.41%   +0.09%     
===========================================
  Files           74       74              
  Lines         1790     1787       -3     
  Branches        78       75       -3     
===========================================
- Hits          1581     1580       -1     
+ Misses         209      207       -2
Impacted Files Coverage Δ
...data/waimak/dataflow/ParallelActionScheduler.scala 100% <100%> (+1.72%) :arrow_up:
...autodata/waimak/dataflow/SequentialScheduler.scala 100% <100%> (ø) :arrow_up:
...coxautodata/waimak/dataflow/DataFlowExecutor.scala 100% <100%> (+3.03%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8633291...851cd94. Read the comment docs.