High traffic on getutxoset

clightning4j / esplora_clnd_plugin

Using @blockstream Esplora web explorer to fetch bitcoin data for clightning

8 stars 11 forks source link

High traffic on getutxoset #39

Closed lvaccaro closed 3 years ago

lvaccaro commented 3 years ago

After funding a channel, lightningd makes a large number of getutxoset requests. Esplora don't support so many requests and plugin could fail.

lvaccaro commented 3 years ago

sorry to ping @darosior , have you test a similar behaviour with sauron plugin?

darosior commented 3 years ago

sorry to ping @darosior , have you test a similar behaviour with sauron plugin?

Sure. Hmm i've never tested Sauron with real-usage load. Maybe did you @cdecker ?

After funding a channel, lightningd makes a large number of getutxoset requests.

Hmm, no: lightningd does not call getutxoset. However we do call gettxout for gossip when we never heard about a scid, to check if the funding transaction was confirmed after a channel cancel, or for some dev commands.

What is Esplora's rate limit ? Also, do you have some logs ? i never experienced such a behaviour, even when syncing gossip.

darosior commented 3 years ago

(gettxout is bitcoind's RPC name, just checked and we call it getutxout in the Bitcoin backend interface)

cdecker commented 3 years ago

getutxout is only called when verifying channels from gossip, while everything else is handled through the internal utxoset table. I expect that funding the channel opened a new connection to the peer, and we started syncing gossip, which leads to the correlation you are observing.

We have a couple of options here:

sync from a lower height so our internal utxoset serves most of those calls
process full blocks in the background and catch up with the chain (being overly optimistic that the outpoint is still unspent is not really an issue)
have a third party service serve snapshots of the utxo.

Trusting the UTXO for channels means the third party can hide existing channels but we're already trusting esplora so why not?

darosior commented 3 years ago

Thanks for the feedback. Trusting Esplora is fine, but i don't think their API provide this snaphost-of-the-utxo-set functionality

lvaccaro commented 3 years ago

https://github.com/lightningamp/esplora_clnd_plugin/pull/48 should fix an error parsing getutxoset and build the json response. This should fix a crash, and previous feedbacks are welcome for high traffic usage