search

btcsuite / btcd

An alternative full node bitcoin implementation written in Go (golang)
https://github.com/btcsuite/btcd/blob/master/README.md
ISC License
6.1k stars 2.31k forks source link

blockchain: accounting error (off-by-one?) in utxo set cache size #2087

Closed Roasbeef closed 4 months ago

Roasbeef commented 6 months ago

Was running some sync tests with various configurations and spotted this upon restart: 

2024-01-01 16:48:11.662 [INF] CHAN: Flushing UTXO cache of 3127 MiB with 1 entries to disk. For large sizes, this can take up to several minutes...

1 entry, yet 3 GB cache size, smells like an accounting error....

Roasbeef commented 6 months ago

cc @kcalvinalvin

kcalvinalvin commented 6 months ago

Will take a look today but my initial assumption is that the 3GB part is how much the preallocated map is taking up.

The actual allocation happens like so:

1: User passes in the acceptable mem usage. 2: Pre-allocate a map based on this. It's usually like half of the mem usage (ex: 8gb of utxocache will alloc ~4gb initially) 3: utxo cache memory usage increases as each utxoentry is cached. The variable sized pkscript places a big part of this memory usage.

Did that happen at the very start? If we're not in headers first, it'll try to "flush" the utxocache. If it was "flushing 1 entry" at the very first start that may be an off by one on the log message It was not an off by one on the log message. On the flush it just had the coinbase tx for block 1 already in the cache.

kcalvinalvin commented 4 months ago

Fixed with #2089