search

btcsuite / btcd

An alternative full node bitcoin implementation written in Go (golang)
https://github.com/btcsuite/btcd/blob/master/README.md
ISC License
6.2k stars 2.36k forks source link

database: investigate replacing leveldb with pebble #2024

Open Roasbeef opened 1 year ago

Roasbeef commented 1 year ago

Today we use a pure Go implementation of leveldb for the main database and relevant indexes. The DB has been able to keep up pretty well over the years, but it's maintained by more or less a single person, and hasn't received any significant commits over the past year or so. As a result, we should consider moving to a better maintained database that can be a drop in replacement for our current usage.

Enter pebble as pure Go KV database inspired by rocksdb (which was inspired by leveldb). One can view this as the 3rd generation of LSM (Log-Structured-Merge) Tree based KV stores. Compared to goleveldb pebble is actively maintained, and has been in used by Cockroachdb in production for ~3 years. The API is very similar to goleveldb, which means we won't need a significant overhaul.

Rather than replace it all at once, we should first add it as a new database option, with no migration path. After enough time, we can add a migration path, t hen eventually deprecate the old database type. At a glance, pebble also appears to be more performant than goleveldb.

kcalvinalvin commented 1 year ago

FWIW leveldb itself hasn't had that much development either. I don't think it necessarily means lack of maintenance.

That being said, I'm not opposed to having pebble as an option. If it ends up being more performant, that's a win. Though maybe have a big warning that pebble itself may not be consensus compatible with leveldb.

Eoous commented 1 year ago

If no enough performance improvement, i think there is no need to replace leveldb with pebble. To replace, we must do a lot of test for pebble to check whether there are internal bugs like missing data or incorrect values when insert or get.

Another point, maybe keepping building pebble means that there are bugs. leveldb is enough stable and safe.