Investigate whether there is really a need for hot -> cold and cold -> cold compactions in the storage layer, or all the hot plus cold under threshold can be compacted in a single go.
This would reduce complexity, and reduce the addition round of IOPs and Spark stage.
Investigate whether there is really a need for
hot -> coldandcold -> coldcompactions in the storage layer, or all the hot plus cold under threshold can be compacted in a single go.This would reduce complexity, and reduce the addition round of IOPs and Spark stage.
https://github.com/CoxAutomotiveDataSolutions/waimak/blob/develop/waimak-storage/src/main/scala/com/coxautodata/waimak/storage/AuditTableFile.scala#L104