martinsumner / leveled

A pure Erlang Key/Value store - based on a LSM-tree, optimised for HEAD requests
Apache License 2.0
355 stars 33 forks source link

Verify Indexes, offline rebuild of keystore #160

Open martinsumner opened 6 years ago

martinsumner commented 6 years ago

See https://github.com/martinsumner/kv_index_tictactree/issues/24

Need to make sure simple leveled functionality exists to verify indexes, and prompt a keystore to be rebuilt whilst a store is offline.

For the first part need to return a folder that will first fold objects (folding over the journal not the ledger) building an AAE tree for index entries of active objects. Then it should fold over all index entries in the Ledger providing an AAE tree to compare, and then compare the AAE trees - and log a warning if they don't compare. Use SnapPreFold = true for both folds, so they can be taken at a consistent point in time.

For the second part need to be able to call book_offlinerebuild(RootPath, ArchivePath), and have that archive the old Ledger, and then open the store with an empty Ledger, and then close again once it is rebuilt.

martinsumner commented 6 years ago

The object fold should fold over the ledger not the Journal, as:

martinsumner commented 6 years ago

It would be good if we could do a partial offline repair here. The index compare fold, would persist a list of broken segment IDs under a GUID, and output this as a log. The offline repair could then be passed the GUID, and fix the index entries that are broken by scanning all the indexes and objects and comparing only those that pass the segment ID filter - and then making only specific index changes.

This would have to be an offline repair, so that issues of concurrent changes to the ledger from the application do not need to be resolved.