search

IntersectMBO / cardano-db-sync

A component that follows the Cardano chain and stores blocks and transactions in PostgreSQL
Apache License 2.0
289 stars 162 forks source link

cardano-db-sync stops/kills cardano-node #1803

Open catad83 opened 1 month ago

catad83 commented 1 month ago

OS Your OS: Ubuntu 24.04

Versions cardano-node 9.1.0 - linux-x86_64 - ghc-8.10 git rev 176f99e51155cb3eaa0711db1c3c969d67438958

cardano-db-sync 13.3.0.0 - linux-x86_64 - ghc-8.10 git revision d21895fec3a9445493b6ff4aa61b8dcb5153de1e

psql (PostgreSQL) 16.3 (Ubuntu 16.3-0ubuntu0.24.04.1)

Build/Install Method cardano-db-sync - build from source

Run method PGPASSFILE=config/pgpass-mainnet cardano-db-sync \ --config $HOME/cardano-src/cardano-node/db-sync-config.json \ --socket-path $HOME/cardano-src/cardano-node/db/node.socket \ --state-dir ledger-state \ --schema-dir schema

Additional context

Problem Report Trying to get a node running with db-sync. Running/starting cardano-db-sync crashes cardano-node. Can somebody point me into a direction on why it's this happening? And maybe how to fix it. Thank you all.

Cardano-Node output:

[cardanon:cardano.node.DiffusionInitializationTracer:Critical:5] [2024-08-02 11:08:06.44 UTC] DiffusionErrored (ExceptionInLinkedThread "ThreadId 174" (ApiMisuse ClosedDBError CallStack (from HasCallStack): prettyCallStack, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/API.hs:350:42 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.API throwApiMisuse, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/State.hs:156:28 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.Impl.State getOpenState, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl.hs:431:49 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.Impl getTipImpl, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl.hs:287:34 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.Impl getTip_, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/API.hs:115:7 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.API getTip, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/API.hs:568:38 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.API getTipSlot, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ChainDB/Impl/Background.hs:154:48 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ChainDB.Impl.Background copyToImmutableDB, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ChainDB/Impl/Background.hs:254:22 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ChainDB.Impl.Background)) [cardanon:cardano.node.ChainDB:Error:5] [2024-08-02 11:08:06.44 UTC] Immutable DB was already closed. Double closing. [cardanon:cardano.node.ChainDB:Info:5] [2024-08-02 11:08:06.44 UTC] Closed Volatile DB. [cardanon:cardano.node.ChainDB:Info:5] [2024-08-02 11:08:06.44 UTC] Closed db with immutable tip at f06990fbfc8b1fe3454fa374002f244d5473014df1ea5ad25222e70a15df6313 at slot 130986321 and tip 0a89b4644d9db9b0827af02b46815c8b1674b897f370c45c8f41754f574a5c6a at slot 131030595

Shutting down.. cardano-node: ExceptionInLinkedThread "ThreadId 174" (ApiMisuse ClosedDBError CallStack (from HasCallStack): prettyCallStack, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/API.hs:350:42 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.API throwApiMisuse, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl/State.hs:156:28 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.Impl.State getOpenState, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl.hs:431:49 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.Impl getTipImpl, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/Impl.hs:287:34 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.Impl getTip_, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/API.hs:115:7 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.API getTip, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ImmutableDB/API.hs:568:38 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ImmutableDB.API getTipSlot, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ChainDB/Impl/Background.hs:154:48 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ChainDB.Impl.Background copyToImmutableDB, called at src/ouroboros-consensus/Ouroboros/Consensus/Storage/ChainDB/Impl/Background.hs:254:22 in ouroboros-consensus-0.20.0.0-c8c9ce6b8e9df299037ea9243bfbe92c8be67d472a67362d1d65d774304e4bc5:Ouroboros.Consensus.Storage.ChainDB.Impl.Background)

cardano-db-sync:

[db-sync-node:Error:255] [2024-08-02 11:08:06.22 UTC] ChainSyncWithBlocksPtcl: AsyncCancelled [db-sync-node.Subscription:Error:251] [2024-08-02 11:08:06.22 UTC] Identity Application Exception: LocalAddress "/home//cardano-src/cardano-node/db/node.socket" MuxError MuxBearerClosed "<socket: 22> closed when reading data, waiting on next header True" [db-sync-node.ErrorPolicy:Warning:80] [2024-08-02 11:08:06.22 UTC] IP LocalAddress "/home//cardano-src/cardano-node/db/node.socket" ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed "<socket: 22> closed when reading data, waiting on next header True"))) 20s 20s [db-sync-node.Subscription:Notice:84] [2024-08-02 11:08:07.22 UTC] Identity Restarting Subscription after 2.014054817s desired valency 1 current valency 0 [db-sync-node.Subscription:Notice:84] [2024-08-02 11:08:15.21 UTC] Identity Starting Subscription Worker, valency 1 [db-sync-node.Subscription:Notice:264] [2024-08-02 11:08:15.21 UTC] Identity Connection Attempt Start, destination LocalAddress "/home//cardano-src/cardano-node/db/node.socket" [db-sync-node.Subscription:Error:264] [2024-08-02 11:08:15.21 UTC] Identity Connection Attempt Exception, destination LocalAddress "/home//cardano-src/cardano-node/db/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (Connection refused) [db-sync-node.ErrorPolicy:Warning:80] [2024-08-02 11:08:15.21 UTC] IP LocalAddress "/home/***/cardano-src/cardano-node/db/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (Connection refused))) 20s 20s [db-sync-node.Subscription:Warning:84] [2024-08-02 11:08:15.24 UTC] Identity Failed to start all required subscriptions [db-sync-node:Error:78] [2024-08-02 11:08:15.38 UTC] runDBThread: AsyncCancelled

kderme commented 1 month ago

I think I've seen this issue myself a couple times. I'm assuming it's a node bug, since db-sync shouldn't be able to kill cardano-node

catad83 commented 1 month ago

I find it odd also. I'm trying to build node 9.0.0 since it states it's compatible with db-sync 13.3.0.0, if it works, it's a bug in node I suppose.