CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Fix for action exceptions causing application to hang #30

Closed vavison closed 6 years ago

vavison commented 6 years ago

Description

Exceptions were being thrown from failed actions before the executor had cleaned up the execution pools, resulting in the application hanging

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Tested locally to verify hanging is no longer happening when an exception is thrown within an action

coveralls commented 6 years ago

Pull Request Test Coverage Report for Build 235


Files with Coverage Reduction New Missed Lines %
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala 1 95.1%
waimak-core/src/main/scala/com/coxautodata/waimak/log/Logging.scala 1 30.77%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlow.scala 1 98.82%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/Storage.scala 1 87.5%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/ParallelActionScheduler.scala 1 96.61%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/FileStorageOps.scala 1 93.1%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlow.scala 1 86.67%
<!-- Total: 7 -->
Totals Coverage Status
Change from base Build 231: 0.2%
Covered Lines: 911
Relevant Lines: 1189

💛 - Coveralls