search

lightninglabs / neutrino

Privacy-Preserving Bitcoin Light Client
MIT License
894 stars 182 forks source link

multi: introduce a batch filter writer #274

Closed ellemouton closed 1 year ago

ellemouton commented 1 year ago

This PR adds a generic BatchWriter which can be used to batch write items to a db. It persists the batch once the size reaches a certain threshold or if a ticker goes off if at least one items is in the batch.

This BatchWriter is then used by the GetCFilter method to persist filters that it gets during a query. This will improve the speed of the query since the BatchWriter is non-blocking.

Performance Improvement

On current master branch, syncing 3200 filters on my local regtest network takes 4m25s With this PR, syncing the same number of filters takes 4.2s

LND CI

LND CI with this PR is being tested here: https://github.com/lightningnetwork/lnd/pull/7788

Chinwendu20 commented 1 year ago

Hello @ellemouton may I please ask how you timed this?

ellemouton commented 1 year ago

@Chinwendu20 - I set up a regtest node & mined a bunch of blocks. Then I connected to it with that script i sent you in slack the other day and would time how long the rescan of those blocks would take. Note that for header syncing (which is what I believe you are working on), you would just want to time that part & not the rescan.

ellemouton commented 1 year ago

@yyforyongyu & @positiveblue - this is ready! The dependant PR has been merged. PTAL 🙏

lightninglabs-deploy commented 1 year ago

@roasbeef: review reminder @positiveblue: review reminder @yyforyongyu: review reminder

Chinwendu20 commented 1 year ago

It seems like this change has not been reflected in btcwallet. I just rebased my branch and I could not run lnd with this new change because of this line of code in btcwallet @ellemouton https://github.com/btcsuite/btcwallet/blob/23eda3cadf717dbf79d191873fc524178492aef9/chain/pruned_block_dispatcher.go#L194

I want to know if there is any way I can bypass this or if I am doing anything wrong. Thank you.

ellemouton commented 1 year ago

@Chinwendu20 see https://github.com/btcsuite/btcwallet/pull/884 and https://github.com/lightningnetwork/lnd/pull/7788