Node fails to create an invoice with timeout

[andrei-21]

The node 034080abcb5c33380ffeb24c94ea5199ce33f5bae497c0c97173c63b0a25e70536 fails to create an invoice or send a payment. All requests fails with

Error calling method Getinfo: RpcError { code: None, message: \"reading response from socket\", data: None }", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "date": "Mon, 29 Apr 2024 18:49:26 GMT", "content-length": "0"} }

There are a lot of error messages in the logs like:

2024-04-30T07:27:43.822364777Z [ERROR] FAILED PRECONDITION: policy failure: get_current_holder_commitment_info: invalid next holder commitment number: 15 != 16
2024-04-30T07:27:45.262164857Z [ERROR] BACKTRACE:
   0: lightning_signer::policy::error::policy_error
             at /home/an/.cargo/registry/src/index.crates.io-6f17d22bba15001f/vls-core-0.11.0/src/policy/error.rs:135:13

...
2024-04-30T07:27:45.267227582Z [ERROR] Signing(Status { code: FailedPrecondition, message: "policy failure: get_current_holder_commitment_info: invalid next holder commitment number: 15 != 16" })

[nepet]

From a first glance this looks like it comes from the channel_reestablish message sent when the node goes back online and the peer tries to reestablish the channel. There could be a few thing that might have gone wrong here. It seems that the commitment counter of the peers is out of sync.

We need to investigate further and will let you know if we need more infos.

[cdecker]

This is an instance of #431