search

lightninglabs / neutrino

Privacy-Preserving Bitcoin Light Client
MIT License
900 stars 183 forks source link

neutrino+query+rescan: improve rescan speed #236

Closed ellemouton closed 1 year ago

ellemouton commented 2 years ago

This PR should probably be split into 3 PR's. Doing it in 1 for now just to get some initial feedback and to better show the progression. The three sections are as follows:

  1. The first 2 commits alter the GetBlock and GetCFilter functions to use the work dispatcher for their queries instead of using the old queryPeers function. The function is now removed bringing us one step closer to removing all query logic from the main package.
  2. The 3rd commit isolates the bottleneck of the GetCFilter function which is persisting filters to the DB. In this commit, this operation is spun off into a goroutine thus allowing the GetCFilter function to return as soon as all the filters are written to the cache.
  3. Finally, the 4th commit ensures that rescan can make use of batch filter fetching by waiting until the header chain is either current or until it is ahead of the specified end height. Before this commit, if rescan is started before the chain is current, then the filters are fetched one by one instead which is what makes things super slow. For example, on local regtest: before this commit it would take 7 seconds to sync 3000 blocks but with this commit, it takes 900ms. (full testnet rescan from local testnet bitcoind took like 5 mins)
ellemouton commented 2 years ago

Thanks for taking a look @positiveblue!!! Will address your comments soon 👍

lightninglabs-deploy commented 2 years ago

@ellemouton, remember to re-request review from reviewers for your latest update

ellemouton commented 2 years ago

!lightninglabs-deploy mute

chappjc commented 2 years ago

Gave it a test on mainnet and there were no more single cfilters requests, just big chunks of blocks, which is great. Not always 1000, but never just 1.

ellemouton commented 2 years ago

awesome! Thanks for testing @chappjc!

losh11 commented 1 year ago

hey guys, is anyone at lightninglabs looking into this? would really appreciate it! resyncing takes forever, and makes user experience really, really bad.

ellemouton commented 1 year ago

Closing - gonna open 3 follow-up PRs 🤓

replaced by https://github.com/lightninglabs/neutrino/pull/273, https://github.com/lightninglabs/neutrino/pull/274 and https://github.com/lightninglabs/neutrino/pull/275