API might overload CLN gRPC API

fusion44 / blitz_api

A management backend for the RaspiBlitz project written in Python / FastAPI

MIT License

19 stars 18 forks source link

API might overload CLN gRPC API #126

Open fusion44 opened 2 years ago

fusion44 commented 2 years ago

refs https://github.com/rootzoll/raspiblitz/issues/3182

This might be the result of a mix of problems:

/list-all-tx being quite inefficient
the WebUI polling /list-all-tx every two seconds
RaspiberryPI being low powered

cstenglein commented 2 years ago

@fusion44 should I poll slower? Like every 10/20s?

fusion44 commented 2 years ago

I'm not sure, need to test whether the polling or even the API really is the problem.

FYI: For every /list-all-tx call I fetch:

every Invoice
every sent payment
copy the Sqlite file of CLN to get the on-chain stuff
sort everything

and, depending on the query params, then send back only a fraction of the data.

Hmm maybe I should just fetch everything from the DB then.

fusion44 commented 2 years ago

LND has an astronomical high CPU usage too. So that's a systemic issue I didn't notice on my dev PC as it is quite a bit faster than a PI.

edit As soon as I close the WebUI browser window and the polling to /list-all-tx stops, the CPU usage stays high. This indicates to /list-all-tx not being the culprit here.

openoms commented 2 years ago

Should find a way to not keep calling functions which return an error. Agree @cstenglein should slow down significantly. Max every 20s as we are still building on the RPi as the main platform. And to be clear this polling only goes on when the webUI is open, right?

fusion44 commented 2 years ago

Yes, the polling in only going on when a browser tab with the WebUI is open. While I agree that we must get away from polling, I actually believe now that the polling is not the root cause of the problem.

I've tested a bit more tonight and the problem persists even as the polling stops.

edit In fact, I've just experienced this problem with no WebUI running at all. It is one of my background tasks which is polling to often. As soon as I kill the API process the CPU usage is returning to normal.

fusion44 commented 2 years ago

@openoms Have you tried to let the node run for a bit? CPU usage tends to get down to normal levels from time to time.

openoms commented 2 years ago

@openoms Have you tried to let the node run for a bit? CPU usage tends to get down to normal levels from time to time.

this https://github.com/fusion44/blitz_api/issues/125#issue-1277958308 looks like that any sent payment record is causing a JSONDecodeError and often leading to this line

E0625 21:44:06.980009804 1907979 fork_posix.cc:76]           Other threads are currently calling into gRPC, skipping fork() handlers

should be rate limited.

openoms commented 2 years ago

Testing Open channel with CLN:

needed to close the WebUI to calm it down.