Blockstream / greenlight

Build apps using self-custodial lightning nodes in the cloud
https://blockstream.github.io/greenlight/getting-started/
MIT License
109 stars 27 forks source link

All API calls return with error "no such file or directory" #481

Closed roeierez closed 1 month ago

roeierez commented 2 months ago

Relai issue: https://github.com/breez/breez-sdk-relai/issues/79

Node Data
***getinfo:***

 status: Internal, message: "No such file or directory (os error 2)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 11 Jul 2024 15:47:54 GMT", "content-length": "0"} }, source: None

***listpeerchannels:***

 status: Internal, message: "No such file or directory (os error 2)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 11 Jul 2024 15:47:59 GMT", "content-length": "0"} }, source: None

***listfunds:***

 status: Internal, message: "No such file or directory (os error 2)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 11 Jul 2024 15:48:04 GMT", "content-length": "0"} }, source: None

***listpayments:***

 status: Internal, message: "No such file or directory (os error 2)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 11 Jul 2024 15:48:10 GMT", "content-length": "0"} }, source: None

***listinvoices:***

 status: Internal, message: "No such file or directory (os error 2)", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Thu, 11 Jul 2024 15:48:15 GMT", "content-length": "0"} }, source: None
cdecker commented 2 months ago

We believe we found the issue that is causing this node to be very slow: either due to an issue registering the node or due to s manual migration from the US region to the EU region, this node has its DB in the EU, but is still marked to execute in the US. This in turn makes the startup very slow, because of the sheet number of queries to start.

We will need to:

cdecker commented 2 months ago

We have identified a couple further nodes that are affected by this issue (<<1%) and are working on fixing up the migration logic to move the nodes to the right places.

cdecker commented 2 months ago

We have fixed up the migration script, allowing us to correct this issue. Now we iterate through the nodes and fix up all the discrepancies.

cdecker commented 1 month ago

All misplaced US nodes have now been migrated to the EU region. Now migrating the EU nodes to the US where necessary. This should be resolved by this evening.

cdecker commented 1 month ago

All misplaces EU nodes with DBs in the US have been moved to the US.