walletdb.mdb grows quickly / unbounded

[bobics]

I'm using bcoin on testnet on SPV node, and seeing unbounded growth in the walletdb.mdb file. I've seen it grow to 30 GB (then ran out of disk space).

I ran a git bisect on bcoin, and this commit seems to be the culprit: https://github.com/bcoin-org/bcoin/commit/bfde70f2ab995e1a20e7878aae42b3daedf6f50b

To Reproduce:

1. Wipe your data folder (this step seems to be important).
2. Start bcoin in SPV mode on testnet.

Result: Look at your walletdb.mdb file. Within a minute it grows to 200 MB, whereas previously it stayed < 1 MB on a brand new empty wallet.

Here's a copy of my local dev conf where I reproduced the issue:

network: testnet
db: lmdb
prefix: /data
use-workers: true
log-file: true
log-level: info
max-files: 8192
api-key: xxxx
http-host: 0.0.0.0

[nodech]

hm.. interesting. I don't see this problem with leveldb, it happens with lmdb.

[nodech]

in https://github.com/bcoin-org/bcoin/commit/bfde70f2ab995e1a20e7878aae42b3daedf6f50b the change that caused the problem for lmdb is db.binding.iterator.

I checked it works fine with db.iterator (coming from AbstractLevelDOWN).

Even if you only initialize and don't use db.binding.iterator, it will cause the same thing.

@chjj

[bucko13]

@bobics - as Nodar pointed out, there are some problems with the lmdb binding. Because of some of these inconsistencies, lmdb support is being deprecated. If you use leveldb though, you should be fine.

[bobics]

@Bucko13 thanks for the update. Mind providing a little more clarity on the decision to deprecate lmdb? Is the main reasoning difficulty due to inconsistencies required in the adapter? It's a bit surprising that lmdb went from being the proposed default, to lmdb being deprecated (and suddenly, from an outside perspective). Especially given that lmdb has some advantages over leveldb, e.g. hot backups and being less prone to corruption.

(I'm not sure if this comment belongs here or in #87)

[bucko13]

Yeah, that's pretty much the reasoning. Bcoin has been built to support the general interface of leveldb, never really using the lmdb specific features anyway. Implementing them to get some of those benefits ends up greatly increasing code complexity for only (what we've evaluated to be) marginal gains. It's been a pretty gradual process of committing most dev time to leveldb and its test coverage. That issue when we considered shifting focus is from over a year ago now, and you can also see all the examples in the docs use leveldb.

Hope that helps answer your question!