Websocket unexpectedly closes on pre-prod, preview or vasil-dev

HirotoShioi commented 2 years ago

What Git revision are you using?

docker image with tag v5.5.5_1.35.3-preprod

What operating system are you using, and which version?

[x] Linux / Ubuntu
[x] Linux / Other
[ ] OSX
[ ] Windows

Describe what the problem is?

When I run, cardano-ogmios with tag v5.5.5_1.35.3-preprod and connect to it with a client (Postman), and do nothing for while, it closes with error 1006 Abnormal Closure: No close frame was received.

スクリーンショット 2022-08-29 17 54 49

Steps to reproduce.

Run ogmios with Docker command: docker run -d -p 1339:1337 --name cardano-ogmios cardanosolutions/cardano-node-ogmios:v5.5.5_1.35.3-preprod
Connect to port 1339 using Postman
Do nothing for 10 seconds or so
Websocket closes with error 1006 Abnormal Closure: No close frame was received.

Weird thing is:

This only occurs to preprod and not on mainnet.
When I send a message immediately after connecting to it, the connection stays open.

I have setup a server that is running both v5.5.5_1.35.3-preprod and v5.5.5_1.35.3-mainnet on server 54.213.247.112 so you can have a look at it.

54.213.247.112:1339 (preprod)

54.213.247.112:1338 (mainnet)

Also, this is the log that's being outputted when I connect to ogmios

preprod

{"severity":"Info","timestamp":"2022-08-29T08:32:10.664567528Z","thread":"16","message":{"Health":{"tag":"HealthTick","status":{"startTime":"2022-08-29T08:19:48.546533797Z","lastKnownTip":{"slot":6078730,"hash":"0037ce04a1047b585839a9ec40dcd09fbf7e22fc0305f1921ac9e7180bd42cfa","blockNo":125489},"lastTipUpdate":"2022-08-29T08:32:10.66444968Z","networkSynchronization":0.99999,"currentEra":"Alonzo","metrics":{"activeConnections":0,"runtimeStats":{"cpuTime":18711579548,"currentHeapSize":428,"gcCpuTime":369041464,"maxHeapSize":642},"sessionDurations":{"max":0,"mean":3010,"min":0},"totalConnections":7,"totalMessages":7,"totalUnrouted":0},"connectionStatus":"connected","currentEpoch":17,"slotInEpoch":376330}}},"version":"v5.5.5"}
{"severity":"Info","timestamp":"2022-08-29T08:32:18.704912814Z","thread":"304","message":{"WebSocket":{"mode":"FullSerialization","tag":"WebSocketConnectionAccepted","userAgent":"User-Agent unknown"}},"version":"v5.5.5"}
{"severity":"Info","timestamp":"2022-08-29T08:32:18.705298589Z","thread":"308","message":{"WebSocket":{"contents":{"agency":"ClientAgency TokPropose","event":"send","tag":"ProposeVersions","versions":["NodeToClientV_10","NodeToClientV_11","NodeToClientV_12","NodeToClientV_13"]},"tag":"WebSocketClient"}},"version":"v5.5.5"}
{"severity":"Info","timestamp":"2022-08-29T08:32:18.705377481Z","thread":"309","message":{"WebSocket":{"contents":{"agency":"ClientAgency TokPropose","event":"send","tag":"ProposeVersions","versions":["NodeToClientV_10","NodeToClientV_11","NodeToClientV_12","NodeToClientV_13"]},"tag":"WebSocketClient"}},"version":"v5.5.5"}
{"severity":"Info","timestamp":"2022-08-29T08:32:18.706133102Z","thread":"308","message":{"WebSocket":{"contents":{"agency":"ServerAgency TokConfirm","event":"receive","tag":"AcceptVersion","version":"\"NodeToClientV_13\""},"tag":"WebSocketClient"}},"version":"v5.5.5"}
{"severity":"Info","timestamp":"2022-08-29T08:32:18.706620383Z","thread":"309","message":{"WebSocket":{"contents":{"agency":"ServerAgency TokConfirm","event":"receive","tag":"AcceptVersion","version":"\"NodeToClientV_13\""},"tag":"WebSocketClient"}},"version":"v5.5.5"}
{"severity":"Error","timestamp":"2022-08-29T08:32:21.707379329Z","thread":"304","message":{"WebSocket":{"exception":"MuxError MuxBearerClosed \"<socket: 18>closed when reading data, waiting on next header True\"","tag":"WebSocketUnknownException"}},"version":"v5.5.5"}
{"severity":"Info","timestamp":"2022-08-29T08:32:21.707456171Z","thread":"304","message":{"WebSocket":{"tag":"WebSocketConnectionEnded","userAgent":"User-Agent unknown"}},"version":"v5.5.5"}

mainnet

 {"severity":"Info","timestamp":"2022-08-29T08:33:28.697969097Z","thread":"393","message":{"WebSocket":{"mode":"FullSerialization","tag":"WebSocketConnectionAccepted","userAgent":"User-Agent unknown"}},"version":"v5.5.5"}
 {"severity":"Info","timestamp":"2022-08-29T08:33:28.698398073Z","thread":"397","message":{"WebSocket":{"contents":{"agency":"ClientAgency TokPropose","event":"send","tag":"ProposeVersions","versions":["NodeToClientV_10","NodeToClientV_11","NodeToClientV_12","NodeToClientV_13"]},"tag":"WebSocketClient"}},"version":"v5.5.5"}
 {"severity":"Info","timestamp":"2022-08-29T08:33:28.698642844Z","thread":"398","message":{"WebSocket":{"contents":{"agency":"ClientAgency TokPropose","event":"send","tag":"ProposeVersions","versions":["NodeToClientV_10","NodeToClientV_11","NodeToClientV_12","NodeToClientV_13"]},"tag":"WebSocketClient"}},"version":"v5.5.5"}
 {"severity":"Info","timestamp":"2022-08-29T08:33:28.699467918Z","thread":"397","message":{"WebSocket":{"contents":{"agency":"ServerAgency TokConfirm","event":"receive","tag":"AcceptVersion","version":"\"NodeToClientV_13\""},"tag":"WebSocketClient"}},"version":"v5.5.5"}
 {"severity":"Info","timestamp":"2022-08-29T08:33:28.699986726Z","thread":"398","message":{"WebSocket":{"contents":{"agency":"ServerAgency TokConfirm","event":"receive","tag":"AcceptVersion","version":"\"NodeToClientV_13\""},"tag":"WebSocketClient"}},"version":"v5.5.5"}

What should be the expected behavior?

The connection should stay open even if I do not send any messages.

Thank you in advance!

HirotoShioi commented 2 years ago

When I delete these configurations, and run it. The node runs as expected.

https://github.com/input-output-hk/cardano-configurations/blob/master/network/preprod/cardano-node/config.json#L16-L20

You can access them via: 54.213.247.112:1337

docker command docker run --name cardano-node-with-config -d -p 1337:1337 -v $(pwd)/cardano-node:/config/cardano-node cardanosolutions/cardano-node-ogmios:v5.5.5_1.35.3-preprod

KtorZ commented 2 years ago

This is probably due p2p being enabled. This is a bug in the networking layer of the node. The issue was fixed some time ago but somehow, the fix was left of the scope of 1.35.3...

At this stage, the only fixes are either:

to disable p2p completely
to make sure that your client send at least one message through the socket immediately after opening it (before 2s).

HirotoShioi commented 2 years ago

@KtorZ Thanks for the insight! How do I disable p2p completely? Due to the time constraint, that is more preferred option for us..!

KtorZ commented 2 years ago

In your node configuration file, you must set 'EnableP2P' to false.

You can have a look at configurations from: https://github.com/input-output-hk/cardano-configurations/tree/master/network

We have disabled p2p by default on all config except those that are explicitly marked with _p2p.

HirotoShioi commented 2 years ago

@KtorZ Thank you! I've diabled P2P and it works. I'll upgrade the node once 1.35.4 is released!

rhyslbw commented 2 years ago

https://github.com/CardanoSolutions/ogmios/issues/249

CardanoSolutions / ogmios