search

apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.19k stars 1.29k forks source link

Forensic tool to identify misoperation on DB #1672

Open xumengpanda opened 5 years ago

xumengpanda commented 5 years ago

In case a buggy application that uses FDB deletes or changes the DB, the application owner may want to know which operation performs the misoperation and when.

A forensic tool that shows all mutations with timestamp will help answer the above questions.

The tool can be built on top of point-in-time restore system which has already parse the backup data and reply the mutations.

xumengpanda commented 4 years ago

A simple way to do this:

Let the new performant restore (aka fast restore) to dump the parsed backup mutations via TraceEvent. The TraceEvent can be consumed and queried by tool like Splunk.

Compared to dumping each ongoing mutation in a cluster into a sink, using backup data for inspecting the historical mutations has the Resource efficiency benefits:

  1. The analysis/auditing can run on demand only when users need them;
  2. We can pre-filter out the not-interested mutations and only dump the specific version-range and key-range mutations.

It has the drawback that users may need to wait for hours to start the analysis. The waiting time can be reduced when the fast restore's performance keeps improving.