search

lni / dragonboat

A feature complete and high performance multi-group Raft library in Go.
Apache License 2.0
4.98k stars 533 forks source link

possible to support https://github.com/bytedance/terarkdb #273

Closed kolinfluence closed 1 year ago

kolinfluence commented 1 year ago

pls support this as storage backend https://github.com/bytedance/terarkdb

lni commented 1 year ago

No, it will not be supported.

First, it is a RocksDB based KV store, storing Raft logs in KV stores is always wasteful. We have a much better storage engine for the logs, it is available at -

https://github.com/lni/dragonboat/tree/master/internal/tan

This engine, called tan, doesn't force you to have keys, why would you need to touch or construct trillions of keys when the leader just want to stream continuous entries to followers. It doesn't do compactions, as the log entries in raft are mostly append only. You also avoid some write amplification when you stop writing logs twice - you don't need to log your log. Its memtable is another redundant component when we already have an in memory log storage inside the raft implementation - inserting into that skiplist based memtable eats a huge chunk of your CPU cycles when you have millions of entries per second.

I'd be willing to be bet that tan is at least 20-30% faster than your suggested library when used for storing raft logs.

Secondly, that suggested library is C++ based.

dioptre commented 1 year ago

Think @kolinfluence you should add it.