search

cardano-foundation / cardano-graphql

GraphQL API for Cardano
Apache License 2.0
262 stars 104 forks source link

Can't start alonzo-purple #576

Closed CyberCyclone closed 3 years ago

CyberCyclone commented 3 years ago

Environment

Git branch: master

Start command:

sudo docker-compose pull && \
sudo NETWORK=alonzo-purple \
API_PORT=3102 \
HASURA_PORT=8092 \
OGMIOS_PORT=1339 \
POSTGRES_PORT=5434 \
docker-compose -p alonzo-purple up -d && \
sudo docker-compose -p alonzo-purple logs -f

Steps to reproduce the bug

Initially starting up I can see these in the logs and it doesn't seem to be pulling any blocks:

cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] before next, messages elided = 1211329771318423
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Chain added block e25b7696d4500fa423f387041a52488739edecefc29d6d13872866b599116064 at slot 0
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Info:32] [2021-08-22 01:44:11.65 UTC] Block fits onto some fork: e25b7696d4500fa423f387041a52488739edecefc29d6d13872866b599116064 at slot 0
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: cbde07f7be9cdefab15e766fa237d7e9dc94f5a989445ae4bbc087e2eecc4888 at slot 0 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Info:32] [2021-08-22 01:44:11.65 UTC] Block fits onto some fork: cbde07f7be9cdefab15e766fa237d7e9dc94f5a989445ae4bbc087e2eecc4888 at slot 0
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Error:32] [2021-08-22 01:44:11.65 UTC] Invalid block e25b7696d4500fa423f387041a52488739edecefc29d6d13872866b599116064 at slot 0: ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationGenesisHashMismatch (GenesisHash {unGenesisHash = 6d58e8c626cb355ac2a4b040e1adb8266753a69f63593274e4558d6bd0c41eac}) (GenesisHash {unGenesisHash = 570b0e27e60daa224088e0d3afce5dce98b62f6280c21b9074797ef61eab64b2})}))
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Info:32] [2021-08-22 01:44:11.65 UTC] Valid candidate genesis (origin)
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Ignoring block: 0085c9a4ee570e8070c178f6a0a1dcc7330df8f473c8974c7d518d42706348f1 at slot 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: e8e33d775b25062b673d672a5d444fb673bb67e288690b2db89da6b0e9fcc6ec at slot 2 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Ignoring block: e8e33d775b25062b673d672a5d444fb673bb67e288690b2db89da6b0e9fcc6ec at slot 2
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: d03da6a61665fdc9b919c9afc8bfa2f9c4e9771d036ed8dfa0639ad8bf438b47 at slot 3 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Ignoring block: d03da6a61665fdc9b919c9afc8bfa2f9c4e9771d036ed8dfa0639ad8bf438b47 at slot 3
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: 1d25ac349b7e245276c7ccfa2b5145186693fd97effa28cba2019d974ea63a38 at slot 4 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Ignoring block: 1d25ac349b7e245276c7ccfa2b5145186693fd97effa28cba2019d974ea63a38 at slot 4
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: 23d06f20b6cecdecdba3f9e14670900ed5b5afb92cc7b1cb5e690fe3d7a3def3 at slot 5 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Ignoring block: 23d06f20b6cecdecdba3f9e14670900ed5b5afb92cc7b1cb5e690fe3d7a3def3 at slot 5
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: aaf3ee7b90280cb59039a469d67cc61b80943f1d6595b0951246c5cb0946db01 at slot 6 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Ignoring block: aaf3ee7b90280cb59039a469d67cc61b80943f1d6595b0951246c5cb0946db01 at slot 6
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: ef20ec7b103ec59648c3d4a429a42ab2950c6fb8adba97e190c0771c5eb381e4 at slot 7 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.65 UTC] Ignoring block: ef20ec7b103ec59648c3d4a429a42ab2950c6fb8adba97e190c0771c5eb381e4 at slot 7
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.65 UTC] Block added to queue: 058690822d4ca688212accc5de1ab99f86ebc7313264174a74a600d92d649c75 at slot 8 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.66 UTC] Ignoring block: 058690822d4ca688212accc5de1ab99f86ebc7313264174a74a600d92d649c75 at slot 8
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.66 UTC] Block added to queue: 9ba0275ec31b56b47977ba9d32c6ca2887d0a694d781ba2f351737bdd0f75710 at slot 9 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.66 UTC] Ignoring block: 9ba0275ec31b56b47977ba9d32c6ca2887d0a694d781ba2f351737bdd0f75710 at slot 9
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.66 UTC] Block added to queue: 9651fc57516f9b5cc7bf5f5bfcbfa9570021dc66d3b5be7629b77b261246c3fa at slot 10 queue size 1
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:32] [2021-08-22 01:44:11.66 UTC] Ignoring block: 9651fc57516f9b5cc7bf5f5bfcbfa9570021dc66d3b5be7629b77b261246c3fa at slot 10
cardano-node-ogmios_1       | [9576f6c0:cardano.node.ChainDB:Debug:80] [2021-08-22 01:44:11.66 UTC] Block added to queue: a945b44b699de313e063db0255498642dd048f7dc61b633c4b6939b78460a454 at slot 11 queue size 1

After bringing the containers down and starting up again I can find this in the logs:

cardano-node-ogmios_1       | [c2a4f747:cardano.node.DnsSubscription:Error:68] [2021-08-22 01:48:39.51 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Application Exception: 18.142.32.1:3001 InvalidBlock (At (Block {blockPointSlot = SlotNo 0, blockPointHash = e25b7696d4500fa423f387041a52488739edecefc29d6d13872866b599116064})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationGenesisHashMismatch (GenesisHash {unGenesisHash = 6d58e8c626cb355ac2a4b040e1adb8266753a69f63593274e4558d6bd0c41eac}) (GenesisHash {unGenesisHash = 570b0e27e60daa224088e0d3afce5dce98b62f6280c21b9074797ef61eab64b2})}))))
cardano-node-ogmios_1       | [c2a4f747:cardano.node.DnsSubscription:Debug:68] [2021-08-22 01:48:39.51 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Closed socket to 18.142.32.1:3001
cardano-node-ogmios_1       | [c2a4f747:cardano.node.ErrorPolicy:Warning:50] [2021-08-22 01:48:39.51 UTC] IP 18.142.32.1:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 0, blockPointHash = e25b7696d4500fa423f387041a52488739edecefc29d6d13872866b599116064})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationGenesisHashMismatch (GenesisHash {unGenesisHash = 6d58e8c626cb355ac2a4b040e1adb8266753a69f63593274e4558d6bd0c41eac}) (GenesisHash {unGenesisHash = 570b0e27e60daa224088e0d3afce5dce98b62f6280c21b9074797ef61eab64b2})}))))))) 200s 200s
cardano-node-ogmios_1       | [c2a4f747:cardano.node.ChainDB:Debug:104] [2021-08-22 01:48:39.65 UTC] New follower was created

I can see many attempts to connect to 'relays.alonzo-purple.dev.cardano.org' but they fail. I'm guessing that the genesis hash doesn't match so it rejects the connection.

What is the expected behavior?

Should be able to connect to 'relays.alonzo-purple.dev.cardano.org' relays and sync blocks.

rhyslbw commented 3 years ago

The network was re-spun yesterday, and the new config picked up in https://github.com/input-output-hk/cardano-configurations/commit/1a79b1e49ffd09a597322b0388b33c40541fd9cd

I'll prepare a release to bundle the new config, but if you manually pull the submodule and build from source it should work.

CyberCyclone commented 3 years ago

The network was re-spun yesterday, and the new config picked up in input-output-hk/cardano-configurations@1a79b1e

I'll prepare a release to bundle the new config, but if you manually pull the submodule and build from source it should work.

Yep, I picked that up too :) . I was waiting for the build to finish to confirm it fixes it.

CyberCyclone commented 3 years ago

@rhyslbw I'll wait till the next bundle has been released.

I seem to be pretty stuck on this, I suspect another module is out of sync. Here's what I've done.

  1. checked out the latest graphql beta branch
  2. checked out master branch on the config directory
  3. checked out master branch in ogmios/server/config/
  4. Follow build instructions.

Stuck on this:

cardano-node-ogmios_1       | [b1494707:cardano.node.DnsSubscription:Debug:147] [2021-08-22 10:19:42.59 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Closed socket to 18.193.243.47:3001
cardano-node-ogmios_1       | [b1494707:cardano.node.DnsSubscription:Debug:56] [2021-08-22 10:19:42.59 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Required subscriptions started
cardano-node-ogmios_1       | [b1494707:cardano.node.ErrorPolicy:Notice:52] [2021-08-22 10:19:42.59 UTC] IP 18.193.243.47:3001 ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = "Parallel connection cancelled", seStack = []}))) 1s
cardano-node-ogmios_1       | [b1494707:cardano.node.DnsSubscription:Error:146] [2021-08-22 10:19:43.29 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Application Exception: 18.142.32.1:3001 MuxError (MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer)) "(recv errored)"
cardano-node-ogmios_1       | [b1494707:cardano.node.DnsSubscription:Debug:146] [2021-08-22 10:19:43.29 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Closed socket to 18.142.32.1:3001
cardano-node-ogmios_1       | [b1494707:cardano.node.ErrorPolicy:Warning:52] [2021-08-22 10:19:43.29 UTC] IP 18.142.32.1:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError (MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer)) "(recv errored)"))) 20s 20s
cardano-node-ogmios_1       | [b1494707:cardano.node.DnsSubscription:Error:144] [2021-08-22 10:19:43.58 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Application Exception: 54.153.11.18:3001 MuxError (MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer)) "(recv errored)"
cardano-node-ogmios_1       | [b1494707:cardano.node.DnsSubscription:Debug:144] [2021-08-22 10:19:43.59 UTC] Domain: "relays.alonzo-purple.dev.cardano.org" Closed socket to 54.153.11.18:3001
cardano-node-ogmios_1       | [b1494707:cardano.node.ErrorPolicy:Warning:52] [2021-08-22 10:19:43.59 UTC] IP 54.153.11.18:3001 ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError (MuxIOException Network.Socket.recvBuf: resource vanished (Connection reset by peer)) "(recv errored)"))) 20s 20s
rhyslbw commented 3 years ago

@CyberCyclone It's likely an incompatibility with cardano-node bundled with cardano-node-ogmios. I've prepared a PR to bump the deps over in Ogmios, which I can then target here. Tests are passing so just need a sanity check from @KtorZ before we can cut a release.

CyberCyclone commented 3 years ago

Just letting you know that I've been able to build alonzo-purple against the current master branch of ogmios. For me this issue is fixed, but I'll let you @rhyslbw close it incase you still wanted it left open.

rhyslbw commented 3 years ago

Thanks for the update. We're looking to cut the release today @CyberCyclone

rhyslbw commented 3 years ago

https://github.com/input-output-hk/cardano-graphql/tree/5-1-stable will be released as 5.1.0

CyberCyclone commented 3 years ago

I've done a fresh install using the released 5.1.0 and alonzo-purple starts perfectly.