CoxAutomotiveDataSolutions / waimak

Waimak is an open-source framework that makes it easier to create complex data flows in Apache Spark.
Apache License 2.0
75 stars 16 forks source link

Feature/single compaction stage #73

Closed vavison closed 5 years ago

vavison commented 5 years ago

Description

Storage compaction no longer performs hot -> cold followed by cold -> cold compactions, instead compacting all of the hot regions plus the cold regions under the configured row threshold in a single go.

This reduces complexity, and removes the additional round of IOPs and Spark stage.

Fixes #71

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

Existing unit tests are still passing

coveralls commented 5 years ago

Pull Request Test Coverage Report for Build 548


Changes Missing Coverage Covered Lines Changed/Added Lines %
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala 19 21 90.48%
<!-- Total: 22 24 91.67% -->
Files with Coverage Reduction New Missed Lines %
waimak-hive/src/main/scala/com/coxautodata/waimak/metastore/HiveDBConnector.scala 1 96.0%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala 1 94.92%
waimak-storage/src/main/scala/com/coxautodata/waimak/storage/StorageActions.scala 1 93.27%
waimak-experimental/src/main/scala/com/coxautodata/waimak/spark/app/SparkApp.scala 1 90.91%
<!-- Total: 4 -->
Totals Coverage Status
Change from base Build 537: -0.01%
Covered Lines: 1255
Relevant Lines: 1567

💛 - Coveralls