Use FoundationDB as an alternative of TiKV for TiDB

[xumengpanda]

Feature Request

FoundationDB (https://github.com/apple/foundationdb) is a deterministic distributed database that supports ACID transactions. It is extremely reliable because it has a deterministic simulator to quickly check its correctness under various interleaving of failure scenarios. One end-to-end test in simulation typically takes less than 30 seconds and covers many interleaving failure scenarios. Once a failure is found, it can be easily reproduced on a single process, which makes development cadence super fast.

FoundationDB is widely used by many big companies, including Snowflake (https://www.snowflake.com/blog/how-foundationdb-powers-snowflake-metadata-forward/) and Wavefront (https://www.wavefront.com/wavefront-foundationdb-open-source-project/).

FoundationDB is also used as the backend storage layer for IBM CouchDB (https://www.youtube.com/watch?v=SjXyVZZFkBg) and eBay JanusGraph (https://www.youtube.com/watch?v=EtB1BPG00PE). The prominent reason why FoundationDB is its reliability, ACID transaction support and performance.

If TiDB can support FoundationDB as a backend storage layer, TiDB will become a very appealing SQL Layer for FoundationDB users. TiDB will also be able to benefit from FoundationDB's deterministic simulation capability.

Note: Previous discussion of SQL Layer support on FDB is at https://forums.foundationdb.org/t/sql-layer-in-foundationdb/94/12

[djshow832]

To support FoundationDB as a storage engine of TiDB, there is some work to do. Here is the list I can think of:

• TiDB From the aspect of TiDB, TiKV needs to conform to the same protocol with TiDB.

  • A RPC client, which conforms to GRPC protocol, such as https://github.com/tikv/grpc-rs. FoundationDB connects to TiDB and PD with this client.
  • TiDB sends 40 kinds of commands to TiKV, including querying, writing, locking, transaction, garbage collecting, etc. See https://github.com/pingcap/kvproto/blob/master/proto/kvrpcpb.proto. FoundationDB needs to process these commands and respond to TiDB in the same way as TiKV does. If an error occurs, FoundationDB should also return the same backoff or error.
  • Among these commands, coprocessor is the most complex part, which significantly improves query performance. The code is in https://github.com/tikv/tikv/tree/master/src/coprocessor.
  • Some commands are storage engine specific, such as pessimistic locks (CmdPessimisticLock), querying historical data (CmdMvccGetByStartTs), and garbage collecting (CmdDeleteRange). Besides, FoundationDB needs to support snapshot and RC isolation as TiKV does.
  • Some features are implemented in TiKV but controlled by TiDB, such as async commit / 1PC (tidb_enable_async_commit), stale read, follower read (tidb_replica_read). These features can be either implemented by FoundationDB or unsupported.
  • AP support: TiDB supports row-store and column-store storage engines and can choose which storage engine to query (by tidb_isolation_read_engines or hints). Obviously, TiDB can always use TP engine if FoundationDB has no plan to support a column-store storage engine.

• PD Both TiKV and TiDB highly rely on PD, which means, FoundationDB needs to be compatible with PD. See https://docs.pingcap.com/tidb/stable/pd-scheduling-best-practices.

  • FoundationDB needs to report various information to PD, including region info, store info, load info. See https://github.com/tikv/tikv/blob/master/components/pd_client/src/client.rs
  • FoundationDB needs to respond to scheduling commands from PD, such as splitting regions, transferring leaders.

• Ecosystem tools TiDB and TiKV support various tools.

  • Data migration and Backup: Foundation needs to support BR / TiCDC / Lightning.
  • Monitoring: FoundationDB needs to report metrics to Grafana (https://github.com/tikv/tikv/tree/master/metrics/grafana) and report data flow to Dashboard KeyViz, if you want to troubleshoot more efficiently.
  • TiKV Control (tikv-ctl): it is used to manage TiKV cluster. FoundationDB needs to support similar tools to manage TiKV cluster in some emergencies, such as restore data when data is corrupted.