Implement Internal Compaction

[clinedome-work]

Add internal compaction to existing live compaction workflow. Previous compaction only eliminated updates from a given segment whose keys exist in a future segment. Here we add the ability to eliminate updates from a given segment whose keys exist later in that same segment.

This is an optimization that, for some types of update profile, could save a significant amount of WAL space and startup time.

Additionally, this is strictly necessary for anyone using the new functionality that will purge old DELETEs from the WAL during compaction.

This addresses issue #117

[kristomasette]

just have a question or two to make sure I'm not seeing double.

Looks good overall though.

[weggert]

I ran a test with this patch against our current Uberstore. The original size of the Uberstore was 41g and the startup was 34 minutes.

The first Internal Compaction against all segments completed in about 1 hour. It reduced the Uberstore size to 34g.

After that a Segment Compaction ran, with the COMPACTION_MAX_DELETE_AGE_HOURS option set. That completed for a few new segments and all merged Segments in another 2 hours.

In the end the Uberstore was reduced from 41g->18g, and the startup time went from 34min->20min.

I verified that the counts of each object type in our repositories stayed the same.

Thanks for implementing this change. The combination of this and removing old Deletes is going to make a big difference for us.

[weggert]

I re-tested with the more precise .isInternallyCompacted logic against our non-compacted Uberstore and everything worked as expected. The Uberstore compacted as expected, and the object counts remained the same.

Thanks for adding the logic.

[weggert]

:+1: Looks good to me.

[kristomasette]

:+1:

[tbarker9comcast]

:+1: