Tracking db implementations

[dangershony]

This issue will track and explore optional db we can use for the fullnode.

Currently for most cases we use leveldb however that is not too reliable (I had a few cases specially when syncing BTC chain of bad data or memory exceptins)

See this comparison: https://mozilla.github.io/firefox-browser-architecture/text/0017-lmdb-vs-leveldb.html

LevelDB is widely noted for being unreliable and databases it manages are prone to corruption

Potential good candidates:

lmdb Lightning Memory-Mapped Database is a very good candidate and I will defo try to make a coind impl of it https://symas.com/lmdb/ Packages: https://github.com/CoreyKaylor/Lightning.NET https://github.com/kwaclaw/KdSoft.Lmdb https://github.com/Spreads/Spreads.LMDB

UltraLiteDB (full C#) A light wait port of litedb https://github.com/rejemy/UltraLiteDB Also suggested optional is to fork litedb and remove some none needed components

I would rather first suggest forking litedb and ripping out everything not needed. LevelDB probably does not have a transaction log, like LiteDB. That probably comes at a fairly large cost. I also think there are some sort of indexing in LiteDB, that comes at cost. And finally the garbage collection is an issue.

Trie (full C#) An improved rewrite of Dbreeze https://github.com/NicolasDorier/DBTrie

But this needs to consider defrag if used for coindb see: https://github.com/NicolasDorier/DBTrie/issues/2

FASTER (full C#) A Microsoft backed fast db https://github.com/microsoft/FASTER There is an implementation in the fullnode but I experienced data corruption (but this was in an earlier version) and it seems to require lots of memory to be efficient.

Bonus: https://eklitzke.org/making-bitcoin-fast-introduction

[NicolasDorier]

Did you bench how Faster compare to LevelDB? Interesting to see.

[dangershony]

ah yeah I will try to add FASTER when I do the next bench