search

Blockstream / greenlight

Build apps using self-custodial lightning nodes in the cloud
https://blockstream.github.io/greenlight/getting-started/
MIT License
109 stars 27 forks source link

Slow API calls #493

Open DeanDonkov opened 1 month ago

DeanDonkov commented 1 month ago

We, a customer of Breez LSP, are having issues on the following endpoints:

Those endpoints result in Breez SDK connection taking upwards of 30 seconds. After communication with their team, it seems the issue might be in Greenlight's nodes.

LOG initializeBreezSDK: 38105ms

cdecker commented 1 month ago

Not quite sure where the slow.startuo in the SDK comes.from. generally speaking using gl-client directly returns getinfo calls in <es. The SDK does a lot of caching, maybe the busy startup sync is delaying user issued commands?

DeanDonkov commented 1 month ago

@cdecker Are you aware of other LSPs using your product? Can you point us to alternatives to Breez LSP? Thanks.

roeierez commented 1 month ago

@DeanDonkov it seems very weird nideInfo (which only use cached data) to take mote than a few milliseconds. Can you share logs of the call so we can drill down here?

martinivv commented 1 month ago

Hi, @roeierez! We're encountering persistent issues with node synchronization and method latency throughout most of the day. What began as sporadic issues 2-3 weeks ago has now escalated into more frequent and disruptive occurrences.

I. Preconditions:

  1. Breez SDK Version: 0.5.2;
  2. Internet Speed: not a limiting factor in this case.

 

II. Issue Details:

1. SDK Initialization Delays: On the first initialization of the SDK, the connection process is significantly delayed and often results in errors.

Error Examples:

[Error: Generic: status: Unavailable, nic:: transport:: Error (Transport, hyper:: Error (Connect, Custom { kind: UnexpectedEof, error: message: were trying to connect t handsheede metadata: Metadata "tls handshake eof" ...]

LOG:: initializeBreezSDK: 300_484ms
[Error: Generic: status: Internal, message: "Too many open files (os error 24)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Fri, 09 Aug 2024 09:23:56 GMT", "content-length": "0"} }, source: None]

 

2. Excessive Method Latency: Some methods, particularly those related to the “node state”, are taking an excessive amount of time to complete.

Example: The receiveOnChain method within the Breez SDK sometimes takes 3-4 minutes or even longer to return a response.

cdecker commented 1 month ago

Please do not ping developers willy nilly, it is rather disruptive, and in this case the Greenlight dev team can't even help as this appears to be entirely handled inside the Breez SDK.

cdecker commented 1 month ago

The Too many open files is indicative of #494 by the way, and may appear spuriously until we fix colliding state updates.