search

lightninglabs / lightning-terminal

Lightning Terminal: Your Home for Lightning Liquidity
MIT License
502 stars 87 forks source link

ResourceExhausted error after fresh install (grpc message > 200MB) #547

Open silenzara opened 1 year ago

silenzara commented 1 year ago

I'm trying to run Terminal Web starting with a fresh install but I run into an issue soon after logging in and "Connecting to Terminal".

On the dashboard I see a few recent forwards appear but nothing after that. Checking the litd logs I see the following error message and right before it (gracefully) shuts down:

2023-05-15 14:30:29.609 [ERR] RMID: Error in interceptor: RPC middleware receive failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (293099810 vs. 209715200)
2023-05-15 14:30:29.609 [ERR] LITD: Received critical error from subsystem, shutting down: RPC middleware receive failed: rpc error: code = ResourceExhausted desc = grpc: received message larger than max (293099810 vs. 209715200)

I suspect this has to do with the amount of data litd tries to load while initializing. As a workaround I recompiled litd with a larger grpc message limit (512MB) but I had to shut it down because lnd started throwing "503, GetWalletInfoErr" errors in other places that make use of the lnd grpc api (I guess there a reason for that 200MB limit 😅)

My node is fairly active with a decent amount of forwards (last week my node crossed the one millionth forward mark for the year) so that's probably relevant.

Environment:

Running litd v0.9.2-alpha with remote lnd v0.16.2-beta. Tried both litd binary release as well as self compiled with go 1.20.4. Both daemons run in their own lxc container.

silenzara commented 1 year ago

Browser logs at the moment litd throws the ResourceExhausted error. Nothing comes after these logs and there are no errors logged before it.

Screenshot_3
silenzara commented 1 year ago

LND logs when the error occurs:

2023-05-15 18:29:31.618 [DBG] RPCP: Exiting middleware run loop for lit-macaroon-firewall-logger
2023-05-15 18:29:31.619 [DBG] RPCP: Removing middleware lit-macaroon-firewall-logger
2023-05-15 18:29:31.619 [INF] NTFN: Cancelling epoch notification, epoch_id=74984
2023-05-15 18:29:31.620 [INF] NTFN: Cancelling epoch notification, epoch_id=74985
2023-05-15 18:29:31.620 [ERR] RPCP: Received an error: rpc error: code = Canceled desc = context canceled, shutting down
2023-05-15 18:29:31.620 [DBG] RPCP: Exiting middleware run loop for lit-account
2023-05-15 18:29:31.620 [INF] INVC: Cancelling invoice subscription for client=44
2023-05-15 18:29:31.620 [ERR] RPCP: Received an error: rpc error: code = Canceled desc = context canceled, shutting down
2023-05-15 18:29:31.620 [DBG] RPCP: Exiting middleware run loop for lit-macaroon-firewall
2023-05-15 18:29:31.620 [DBG] RPCP: Removing middleware lit-account
2023-05-15 18:29:31.620 [DBG] RPCP: Removing middleware lit-macaroon-firewall
2023-05-15 18:29:31.620 [ERR] RPCP: Received an error: rpc error: code = Canceled desc = context canceled, shutting down
2023-05-15 18:29:31.620 [DBG] RPCP: Exiting middleware run loop for lit-privacy-mapper
2023-05-15 18:29:31.620 [DBG] RPCP: Removing middleware lit-privacy-mapper
2023-05-15 18:29:31.620 [ERR] CHAC: Received an error: rpc error: code = Canceled desc = context canceled, shutting down
2023-05-15 18:29:31.620 [INF] NTFN: Cancelling epoch notification, epoch_id=74986
2023-05-15 18:29:31.625 [DBG] RRPC: htlc event stream cancelled
2023-05-15 18:29:31.625 [INF] INVC: Cancelling invoice subscription for client=45
silenzara commented 1 year ago

Entire litd log: https://pastebin.mozilla.org/2HVpg1sC