search

IntersectMBO / cardano-node

The core component that is used to participate in a Cardano decentralised blockchain.
https://cardano.org
Apache License 2.0
3.07k stars 720 forks source link

[BUG] - Node shuts down after TrServerError #6016

Open linconvidal opened 1 month ago

linconvidal commented 1 month ago

External

Area Other Any other topic (Delegation, Ranking, ...).

Summary I had a node running on preview for several days, but today I noticed it had shut down. It appears to be related to a connection issue, as it took less than 15 seconds from chain extension to shutdown. Restarting the node resolved the issue.

Steps to reproduce I have not been able to reproduce the issue yet. I’m recording it here to bring attention to a potential problem.

Expected behavior If there is a disconnection issue, the node should attempt to reconnect for more than just a few seconds before shutting down.

System info (please complete the following information):

Screenshots and attachments

cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Notice:33] [2024-10-10 08:55:01.19 UTC] Chain extended, new tip: b36626d4ec63cdf6589e9b9c1c53de0a65f9d78770153c2c044e5f7b6c92625c at slot 61894501
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Notice:33] [2024-10-10 08:55:17.18 UTC] Chain extended, new tip: 686ecd4d9b906994f0a14886dfe40510b4fc94af7370654f3779f4d407a9699b at slot 61894517
cardano-node-1  | [0dd6344b:cardano.node.Mempool:Info:42765] [2024-10-10 08:55:29.50 UTC] fromList [("kind",String "TraceMempoolAddedTx"),("mempoolSize",Object (fromList [("bytes",Number 1295.0),("numTxs",Number 1.0)])),("tx",Object (fromList [("txid",String "b1b1ba29")]))]
cardano-node-1  | [0dd6344b:cardano.node.Server:Critical:59] [2024-10-10 08:55:33.31 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.Server:Notice:51] [2024-10-10 08:55:33.31 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.PeerSelection:Error:57] [2024-10-10 08:55:33.32 UTC] TraceOutboundGovernorCriticalFailure AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Critical:67] [2024-10-10 08:55:33.32 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Notice:62] [2024-10-10 08:55:33.32 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:62] [2024-10-10 08:55:33.32 UTC] TrShutdown
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:232] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 3, inboundConns = 3, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:240] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:242] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:1124] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:259] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:1121] [2024-10-10 08:55:33.32 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:51] [2024-10-10 08:55:33.33 UTC] TrShutdown
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48294] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 57, unidirectionalConns = 0, inboundConns = 3, outboundConns = 56})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49711] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 55, unidirectionalConns = 0, inboundConns = 3, outboundConns = 54})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:40716] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 50, unidirectionalConns = 0, inboundConns = 3, outboundConns = 49})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48175] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 49, unidirectionalConns = 0, inboundConns = 3, outboundConns = 48})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48328] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 47, unidirectionalConns = 0, inboundConns = 3, outboundConns = 46})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45276] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 46, unidirectionalConns = 0, inboundConns = 3, outboundConns = 45})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47097] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 43, unidirectionalConns = 0, inboundConns = 3, outboundConns = 42})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48433] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 43, unidirectionalConns = 0, inboundConns = 3, outboundConns = 42})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48689] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 42, unidirectionalConns = 0, inboundConns = 3, outboundConns = 41})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47726] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 42, unidirectionalConns = 0, inboundConns = 3, outboundConns = 41})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49234] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 36, unidirectionalConns = 0, inboundConns = 3, outboundConns = 35})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48742] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 36, unidirectionalConns = 0, inboundConns = 3, outboundConns = 35})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:38526] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 35, unidirectionalConns = 0, inboundConns = 3, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47319] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 34, unidirectionalConns = 0, inboundConns = 2, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:43789] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 34, unidirectionalConns = 0, inboundConns = 2, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:46331] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 34, unidirectionalConns = 0, inboundConns = 2, outboundConns = 34})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47698] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 29, unidirectionalConns = 0, inboundConns = 2, outboundConns = 29})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:38786] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 29, unidirectionalConns = 0, inboundConns = 2, outboundConns = 29})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49733] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 28, unidirectionalConns = 0, inboundConns = 2, outboundConns = 28})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49712] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 28, unidirectionalConns = 0, inboundConns = 2, outboundConns = 28})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49681] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 26, unidirectionalConns = 0, inboundConns = 2, outboundConns = 26})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49131] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 26, unidirectionalConns = 0, inboundConns = 2, outboundConns = 26})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49401] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 26, unidirectionalConns = 0, inboundConns = 2, outboundConns = 26})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47213] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 25, unidirectionalConns = 0, inboundConns = 2, outboundConns = 25})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:37792] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 25, unidirectionalConns = 0, inboundConns = 2, outboundConns = 25})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45777] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 24, unidirectionalConns = 0, inboundConns = 2, outboundConns = 24})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47679] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 24, unidirectionalConns = 0, inboundConns = 2, outboundConns = 24})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:37940] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 22, unidirectionalConns = 0, inboundConns = 2, outboundConns = 22})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49753] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48729] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48467] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:40209] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:46268] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45061] [2024-10-10 08:55:33.33 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 21, unidirectionalConns = 0, inboundConns = 2, outboundConns = 21})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48361] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 18, unidirectionalConns = 0, inboundConns = 2, outboundConns = 18})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49126] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 18, unidirectionalConns = 0, inboundConns = 2, outboundConns = 18})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47727] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 2, duplexConns = 18, unidirectionalConns = 0, inboundConns = 2, outboundConns = 18})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48678] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 16, unidirectionalConns = 0, inboundConns = 1, outboundConns = 16})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:38191] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 16, unidirectionalConns = 0, inboundConns = 1, outboundConns = 16})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45673] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 16, unidirectionalConns = 0, inboundConns = 1, outboundConns = 16})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42027] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 10, unidirectionalConns = 0, inboundConns = 1, outboundConns = 10})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49434] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 9, unidirectionalConns = 0, inboundConns = 1, outboundConns = 9})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:44901] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 9, unidirectionalConns = 0, inboundConns = 1, outboundConns = 9})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42978] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:45693] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49504] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49710] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 1, duplexConns = 8, unidirectionalConns = 0, inboundConns = 1, outboundConns = 8})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:46636] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 6, unidirectionalConns = 0, inboundConns = 0, outboundConns = 6})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42040] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 5, unidirectionalConns = 0, inboundConns = 0, outboundConns = 5})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48897] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 5, unidirectionalConns = 0, inboundConns = 0, outboundConns = 5})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42821] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 5, unidirectionalConns = 0, inboundConns = 0, outboundConns = 5})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48419] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 3, unidirectionalConns = 0, inboundConns = 0, outboundConns = 3})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:42783] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 3, unidirectionalConns = 0, inboundConns = 0, outboundConns = 3})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49687] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 2, unidirectionalConns = 0, inboundConns = 0, outboundConns = 2})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:43928] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49141] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:48677] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49171] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:40635] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:49474] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.ConnectionManager:Info:47370] [2024-10-10 08:55:33.34 UTC] TrConnectionManagerCounters (ConnectionManagerCounters {fullDuplexConns = 0, duplexConns = 0, unidirectionalConns = 0, inboundConns = 0, outboundConns = 0})
cardano-node-1  | [0dd6344b:cardano.node.DiffusionInitializationTracer:Critical:5] [2024-10-10 08:55:33.34 UTC] DiffusionErrored thread killed
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Info:5] [2024-10-10 08:55:33.35 UTC] Closed Immutable DB.
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Info:5] [2024-10-10 08:55:33.35 UTC] Closed Volatile DB.
cardano-node-1  | [0dd6344b:cardano.node.ChainDB:Info:5] [2024-10-10 08:55:33.35 UTC] Closed db with immutable tip at d7941d06ad23506f4e858f1a4300996763c8ef48e4368bcd048389a6f9520d01 at slot 61882724 and tip 686ecd4d9b906994f0a14886dfe40510b4fc94af7370654f3779f4d407a9699b at slot 61894517
cardano-node-1  | 
cardano-node-1  | Shutting down..
cardano-node-1  | cardano-node: thread killed
carbolymer commented 1 month ago

cardano-node-1 | [0dd6344b:cardano.node.DiffusionInitializationTracer:Critical:5] [2024-10-10 08:55:33.34 UTC] DiffusionErrored thread killed

Have you checked if it was not an operating system fault, in system logs for example? Have you checked if you have not run out of memory?

coot commented 3 weeks ago

The relevant section of the log doesn't show where the problem originated from:

cardano-node-1  | [0dd6344b:cardano.node.Server:Critical:59] [2024-10-10 08:55:33.31 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.Server:Notice:51] [2024-10-10 08:55:33.31 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.PeerSelection:Error:57] [2024-10-10 08:55:33.32 UTC] TraceOutboundGovernorCriticalFailure AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Critical:67] [2024-10-10 08:55:33.32 UTC] TrServerError AsyncCancelled
cardano-node-1  | [0dd6344b:cardano.node.LocalServer:Notice:62] [2024-10-10 08:55:33.32 UTC] TrServerStopped
cardano-node-1  | [0dd6344b:cardano.node.LocalConnectionManager:Info:62] [2024-10-10 08:55:33.32 UTC] TrShutdown

asynchronous exceptions mean that some other thread decided to kill the server. Could you share full logs?