Closed guggero closed 3 years ago
If a neutrino node has a database file that is partially synced, this PR breaks it, so would need a migration.
Good point. I added a fallback to the root bucket like @halseth suggested and also added a test for it.
I think it would pay dividends to implement parallel header download (#71) instead.
Do you mean instead of merging this PR? I think we can still implement the parallel header download and get benefits from both improvements.
If a neutrino node has a database file that is partially synced, this PR breaks it, so would need a migration.
Good point. I added a fallback to the root bucket like @halseth suggested and also added a test for it.
I think it would pay dividends to implement parallel header download (#71) instead.
Do you mean instead of merging this PR? I think we can still implement the parallel header download and get benefits from both improvements.
both improvements makes sense, disregard earlier comment will re-review
Fixes https://github.com/lightninglabs/neutrino/issues/196.
With this PR we store the block index keys (hash->height) in sub- buckets with the first two bytes of the hash as the bucket name. Storing a large number of keys in the same bucket has a large impact on memory usage in bbolt if small-ish batch sizes are used (the b+ tree needs to be copied with every resize operation). Using sub buckets is a compromise between memory usage and access time. 2 bytes (=max 65535 sub buckets) seems to be the sweet spot (-50% memory usage, +30% access time). We take the bytes from the beginning of the byte-serialized hash since all Bitcoin hashes are reverse-serialized when displayed as strings. That means the leading zeroes of a block hash are actually at the end of the byte slice.
As the benchmarks below show, the 2 byte prefix seems to be the sweet spot between memory usage, access speed and DB file size.
I also looked at other ways of reducing the memory footprint of the
bbolt
based index. The main culprit seems to be the relatively small batch size of 2k blocks per update. That number is limited by how many blocks a peer serving block headers is serving in one message and cannot easily be increased. The only other way to increase the DB write batch size is to add a second level cache. But that would require more refactoring and could lead to possible de-synchronization of the index and the actual block file.Benchmarks
All results are retrieved by running:
System:
Test 1: control, no changes
Benchmark output
Max DB file size: ~63 MB
Memory usage (
go tool pprof mem.out
->top
):Test2: 1 byte sub bucket
Benchmark output
Max DB file size: ~64 MB
Memory usage (
go tool pprof mem.out
->top
):Test 3: 2 byte sub bucket
Benchmark output
Max DB file size: ~69 MB
Memory usage (
go tool pprof mem.out
->top
):Test 4: 3 byte sub bucket
Benchmark output
Max DB file size: ~106 MB
Memory usage (
go tool pprof mem.out
->top
):