Cluster wide Bitcask merge synchronization

[cdahlqvist]

Large amounts of concurrent Bitcask merging activity across a cluster can have a significant impact on latencies and performance. One of the recommended ways to get around this is to use staggered, non-overlapping merge windows, resulting in at most one node performing merging at any point in time. A potential issue with this approach is however that it means merging on each node occurs less frequently, leading to a significant increase in disk usage between windows if a lot of data is inserted/updated.

Given that the aim of this approach is to ensure concurrent merging is kept to a minimum, would it make sense to introduce a global Bitcask merge coordinator that automatically serialises merges across the cluster?

Each individual node could still observe any configured merge windows, but the coordinator would ensure that merges are not performed concurrently. As concurrent merges are undesirable but not necessarily causes problems, a best effort approach might be sufficient.

[evanmcc]

I don't think that this is a good idea. The mechanism would be complicated and likely fragile. I think that the best way to address this is to reduce the impact of merging on any given node to the point where we no longer care that it's going on.

Something like this might be possible in the future when there is consistent cluster metadata, but hopefully we'll have solved the core problem by then.

[bsparrow435]

How about a warning in the logs that nodes share merge windows? Since a merge_window is a global bitcask setting it should be a simple rpc:multicall to get the merge windows from all nodes and comparing them is easy.

[evanmcc]

but wouldn't you get a warning by default? I'd rather put the effort into making merges suck less, really.