CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.

Apache License 2.0

75 stars 16 forks source link

Description

This PR introduces behaviour to clean up the temporary folder (if given) after a flow has been executed.

This behaviour can be disabled by setting the configuration property spark.waimak.dataflow.removeTempAfterExecution to false.

In all cases, the folder will not be deleted if flow execution fails.

Fixes #70

Type of change

Please delete options that are not relevant.

[ ] Bug fix (non-breaking change which fixes an issue)
[X] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

Unit tests

Requires docs change

Files with Coverage Reduction	New Missed Lines	%
waimak-hive/src/main/scala/com/coxautodata/waimak/metastore/HiveDBConnector.scala	1	96.0%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkDataFlow.scala	1	85.51%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala	1	95.38%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkInterceptors.scala	1	94.44%
waimak-core/src/main/scala/com/coxautodata/waimak/log/Logging.scala	1	30.77%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/DataFlow.scala	1	93.38%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/ParallelActionScheduler.scala	1	96.3%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/StorageActions.scala	1	94.23%
waimak-core/src/main/scala/com/coxautodata/waimak/dataflow/spark/SparkActions.scala	1	70.37%
waimak-experimental/src/main/scala/com/coxautodata/waimak/spark/app/SparkApp.scala	1	90.91%
<!--	Total:	12	-->

Files with Coverage Reduction

New Missed Lines

96.0%

85.51%

95.38%

94.44%

30.77%