Cannot trade with dlc channel in state RenewOffered

[holzeis]

• Rollover timed out.
• No new rollover was attempted.
• After expiry the position did not get closed automatically.
• Close position button is in closing

Trying to buy $1 resulted in the reported error.

[holzeis]

It seems the app never received the RenewOffer.

[holzeis]

I checked the logs, the coordinator wasn't able to store the RenewOffer dlc message, thus it is never sent in the first place, and because it wasn't stored to the database it wasn't retried on reconnects.

{"timestamp":"2024-06-14T15:18:30.520400301Z","level":"ERROR","fields":{"message":"Failed to store dlc message. timed out waiting for connection","peer":"02aa6fc81e341d69b36174365644a3560bfcc51ce0fb9bb67d93712bc54117cbb8"},"target":"coordinator::dlc_handler"}

It looks like we are too inefficient with our connection usage. Haven't checked yet, but if we use the default values 10 connections can be requested in parallel from the postgresql database. If all of them a used and not freed up in time, the client will time out.

Maybe related to some inefficiency on the rollover implementation.

[holzeis]

This problem seems to be happening frequently

https://itchysats.grafana.net/explore?schemaVersion=1&panes=%7B%2291e%22:%7B%22datasource%22:%22grafanacloud-logs%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22grafanacloud-logs%22%7D,%22editorMode%22:%22builder%22,%22expr%22:%22%7Bcontainer%3D%5C%22coordinator%5C%22%7D%20%7C%3D%20%60timed%20out%20waiting%20for%20connection%60%20%7C%20json%22,%22queryType%22:%22range%22%7D%5D,%22range%22:%7B%22from%22:%22now-30d%22,%22to%22:%22now%22%7D%7D%7D&orgId=1

[bonomat]

If all of them a used and not freed up in time, the client will time out.

What's the timeout?

[holzeis]

If all of them a used and not freed up in time, the client will time out.

What's the timeout?

https://github.com/get10101/10101/issues/2629#issuecomment-2172701869

[holzeis]

Closing this issue as the underlying issue is #2629 which resulted in the dlc message not getting stored to the database thus the rollover never happened. I rolled back the channel state on the coordinator and rerun the rollover which worked.