lbryio / lbrycrd

The blockchain that provides the digital content namespace for the LBRY protocol
https://lbry.com
MIT License
2.58k stars 178 forks source link

Memory improvements, claimtrie revamp #276

Closed BrannonKing closed 5 years ago

BrannonKing commented 5 years ago

This PR addresses the following issues:

Fixes #106 (Reimplement claimtrie without so many empty nodes, save RAM) Fixes #108 (Separate data structure and data operations) Fixes #133 (Trie serialization failure) Fixes #138 (Rename nCurrentHeight) Fixes #145 (Rename CClaimTrieCache) Fixes #158 (Properly delete CClaimTrieNode)

It also includes these enhancements:

  1. BLOCK_HASH is no longer needlessly stored in the claimtrie DB. I had a theory that this was masking places that we should have been comparing the claimTrieHash, which theory proved correct.
  2. An issue with takeover height computation was detected and (temporarily) worked around.
  3. cachedTakeoverHeights was removed; we now use the takeoverHeight field on the nodes in the cache.
  4. The getQueue* functions now re-use code.
  5. effectiveAmount is updated when reading from disk (we should now ensure that it's not being needlessly computed elsewhere).

Although the commits have been squashed, much of this work was done by @bvbfan .

We experimented with keeping empty nodes truly empty -- allocating claim space for them as needed. This saved less than 10MB of RAM and complicated the code a bit; we ditched it. We also experimented with a similar approach where empty node hashes were kept in a separate map. This also was too complicated to get right.