Spill transaction log data onto storage servers

[etschannen]

When enough data builds up on the transaction logs that we can no longer keep all of the unpopped mutations in memory, we need to write the mutations to disk. We call this process "spilling" the mutation from memory to disk.

Spilling puts an additional burden on the transaction logs that reduces their throughput significantly. The goal of this project is to have storage servers pull data from unpopped tags instead of letting the data stay on the transaction logs. Worst case, this will 2x increased write load on the storage servers. This plan also means we can spill for much longer without running out of disk space, and the disked used by the transaction logs can be sized to exactly fit memory.

There are two parts to implementing this. The first is a mechanism for telling the system to spill a tag to a set of storage servers:

• If no data is currently being spilled, set the startVersion to 0, otherwise use versionstampedkey as the startVersion
• The key also contains the tag that is being spilled
• The value is set to a vector of storage server IDs that will have the data
• ApplyMetadataMutation will take the message with the tags for those storage servers and private mutations that trigger the storage server to start copying data from that tag
• The spilled data is not stored in the main database keyspace so that it is not subject to data distribution moving it
• This means it is stored with a /xff/xff/xff prefix in the btree, and requires a new endpoint on the storage server to query the data
• All of the tlogs keep the map of where data is being spilled
• Peek requests in version ranges that are spilled provide the storage server ids
• The client caches a map of storage server ids to storage server interfaces, load balance is used to request version ranges
• pop requests are routed to a new endpoint on the storage servers

The second part is adding a component to data distribution which will spill tags for failed storage servers or regions.

[etschannen]

The new transaction log spilling mechanism in 6.1 seems to be performing well enough that this feature is not needed in the short term.