CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Feature/action logging in stages ui #6

Closed alexjbush closed 6 years ago

alexjbush commented 6 years ago

Description

Log stage information to the spark ui.

Type of change

Please delete options that are not relevant.

May possibly break initialisation of Sequential Executor if it has not been built through Waimak builder. Easy to update existing code.

How Has This Been Tested?

Unit test to check actions are logged. Had to change existing unit tests as logLabel output has changed, so using description now (as previously GUID was trimmed off logLabel).

coveralls commented 6 years ago

Pull Request Test Coverage Report for Build 22


Changes Missing Coverage Covered Lines Changed/Added Lines %
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkActions.scala 9 20 45.0%
<!-- Total: 20 31 64.52% -->
Files with Coverage Reduction New Missed Lines %
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala 1 94.29%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/SequentialDataFlowExecutor.scala 1 94.12%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/StorageActions.scala 1 88.89%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkInterceptors.scala 1 94.44%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkActions.scala 1 69.15%
waimak-rdbm-ingestion/src/main/scala/com/coxautodata/waimak/rdbm/ingestion/RDBMIngestionActions.scala 1 97.3%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlow.scala 2 82.93%
<!-- Total: 8 -->
Totals Coverage Status
Change from base Build 21: 0.5%
Covered Lines: 716
Relevant Lines: 1006

💛 - Coveralls