eBay / akutan

A distributed knowledge graph store
Apache License 2.0
1.65k stars 107 forks source link

Log prefix truncation #3

Open superfell opened 5 years ago

superfell commented 5 years ago

When using the logspec client, the cluster needs to regularly calculate a safe truncation point, and tell the log store to truncate the prefix of the log to that index.

The API server can consult with all the views to determine the smallest safe point across the cluster. Either the API tier can then issue the delete, or for more complex deployments where a single log is used across multiple DCs, and each DC has its own beam cluster, you'd probably want this truncation to be managed by a separate control plane.