search

cryptosharks131 / lndg

Lite GUI web interface to analyze lnd data and leverage the backend database for automation tools around rebalancing and other basic maintenance tasks.
MIT License
232 stars 49 forks source link

More efficient forwards updates #404

Closed djkazic closed 9 hours ago

djkazic commented 19 hours ago

Today, the updates of forwards is done with a fixed start_date and incrementing the index_offset. While this technically works, it places an unnecessary amount of strain on the backing LND instance. This PR changes the logic to make use of incrementing index_offset, but only if a start_date window is exhausted. This results in a large speed-up for syncing forward events in nodes that either use a slower DB backend (like kvdb_postgres) or if there are just lots of forwarding events (200k+).

djkazic commented 19 hours ago

This patch was tested on my node as well as Nodelou and RecklessApotheosis.

Filouman commented 18 hours ago

This update to jobs.py not only greatly increased import speed of initial forwarding data in our tests, but also speeds up subsequent fwdinghistory calls that LNDg makes (every 20 seconds by default), which is a HUGE QOL improvement. Running this patch improved page loading times massively and completely eliminated the frequent timeouts I had been experiencing.

warioishere commented 15 hours ago

tested also here on a big node with a bbolt db of arround 23gb. Huge speed up in almost all tabs, slighlty reduced CPU usage on the VM, no more server error opening channel performance tab

great job guys!

recklessnode commented 14 hours ago

Yes, this is an incredible performance increase on all tabs, and on rebalances it seems. Great work @djkazic and @Filouman this is a great optimization and demonstrates the importance of well structured queries! Thanks for this awesome fix.