CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Fix for parallel scheduler hanging on certain types of exceptions #80

Closed alexjbush closed 5 years ago

alexjbush commented 5 years ago

Description

Rework handling of exceptions to take into account fatal exceptions. This fixes the issue of some types of exceptions causing flows to hang indefinitely.

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Unit tests

codecov-io commented 5 years ago

Codecov Report

Merging #80 into develop will increase coverage by 0.04%. The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff            @@
##           develop    #80      +/-   ##
=========================================
+ Coverage    81.95%    82%   +0.04%     
=========================================
  Files           56     56              
  Lines         1585   1589       +4     
  Branches        66     77      +11     
=========================================
+ Hits          1299   1303       +4     
  Misses         286    286
Impacted Files Coverage Δ
...data/waimak/dataflow/ParallelActionScheduler.scala 98.27% <100%> (+0.12%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 3e80947...496772f. Read the comment docs.