Pebble is a high performant
backend from Cochroachdb, used by default in Cluster:
Proven to work well on very large pinsets.
Best disk-usage compared to the rest. No need to trigger GC cycles for space reclaim.
Performance and memory usage seems on par with Badger3, and behaves better than Badger on both counts.
Behaves correctly with default settings but we bump them up a bit.
0-delay startup times, even with very large amounts of data.
Options support compression (we chose to leave it enabled by default).
The Pebble project is officially alive and maintained.
Pebble only runs on 64-bit architectures.
One key difference with Badger3 is that Pebble stores keys and values
together and any lookup for a key will also read the values, while Badger3
can store keys and values separately (i.e. keys only in the index, which can
be loaded onto memory when small enough).
After changing leveldb store to pebble store, the speed of GC has increased by at least dozens of times. So it's not flatfs that's to blame for slow GC, it's leveldb. I also tried leveldb and pebble as blockstore, but the CPU and memory usage is unacceptable.
Right now, to use go-ds-pebble one needs to build external plugin.
[ ] (TBD, sounds like) do we want to allow flatfs (for blocks)+pebble for pins? if so, we need separate profile for applying it only as leveldb replacement, and separate one for /blocks (blockstore)
Checklist
Description
Summary
Include pebble as built-in plugin. It provides meaningful alternative to leveldb as the datastore, and may be better than badger1 as well.
Background
It is 2024, and we still only have flatfs, leveldb and necromancy-level badgerv1 (!) as datastore options.
We got positive feedback about pebble, some examples:
https://ipfscluster.io/documentation/guides/datastore/
https://github.com/ipfs/go-ds-pebble/issues/29:
Right now, to use go-ds-pebble one needs to build external plugin.
Proposed change
Include https://github.com/ipfs/go-ds-pebble in standard kubo build, just like we do with legacy badger1.
This will
TODO
/blocks
(blockstore)