search

IntersectMBO / cardano-node

The core component that is used to participate in a Cardano decentralised blockchain.
https://cardano.org
Apache License 2.0
3.06k stars 721 forks source link

[BUG] - node stuck at slot 21365 block number 20335 ValidationError/SubscriberError/ErrorPolicySuspendConsumer #1724

Closed tanzeelrana closed 4 years ago

tanzeelrana commented 4 years ago

External

Summary Following the instructions at https://cardano-community.github.io/guild-operators/#/Build/node-cli

See my commands history

Screen Shot 2020-08-18 at 4 12 31 PM

Steps to reproduce

sudo ufw allow proto tcp from any to any port 22 
sudo ufw allow proto tcp from any to any port 6000
sudo ufw enable
mkdir "$HOME/tmp";cd "$HOME/tmp"
curl -sS -o prereqs.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/prereqs.sh

chmod 755 prereqs.sh
./prereqs.sh
 cd $CNODE_HOME
. "${HOME}/.bashrc"
cd $CNODE_HOME

curl -sL -o $CNODE_HOME/files/byron-genesis.json https://hydra.iohk.io/job/Cardano/iohk-nix/cardano-deployment/latest-finished/download/1/testnet-byron-genesis.json

curl -sL -o $CNODE_HOME/files/genesis.json https://hydra.iohk.io/job/Cardano/iohk-nix/cardano-deployment/latest-finished/download/1/testnet-shelley-genesis.json

curl -sL -o $CNODE_HOME/files/topology.json https://hydra.iohk.io/job/Cardano/iohk-nix/cardano-deployment/latest-finished/download/1/testnet-topology.json

curl -sL -o $CNODE_HOME/files/config.json https://raw.githubusercontent.com/cardano-community/guild-operators/master/files/ptn0-combinator.json

cd ~/git
git clone https://github.com/input-output-hk/cardano-node
cd cardano-node
git fetch --tags --all
git checkout tags/1.18.0
git pull
git pull origin master 
echo -e "package cardano-crypto-praos\n  flags: -external-libsodium-vrf" > cabal.project.local
$CNODE_HOME/scripts/cabal-build-all.sh
cd $CNODE_HOME 

./cnode.sh

My node runs up to the end of 1st epoch and then peers disappear and it gets stuck

Screen Shot 2020-08-18 at 4 06 10 PM

Expected behavior It should continue to validate as expected

System info (please complete the following information):

root@ubuntu-s-6vcpu-16gb-fra1-01:~/git/cardano-node# cardano-cli version
cardano-cli 1.18.0 - linux-x86_64 - ghc-8.6
git rev 9e06d6f9a293a9656bf6e31b7e5d0a9f498163d1
root@ubuntu-s-6vcpu-16gb-fra1-01:~/git/cardano-node# cardano-node version
cardano-node 1.18.0 - linux-x86_64 - ghc-8.6
git rev 9e06d6f9a293a9656bf6e31b7e5d0a9f498163d1

Additional context

I went in to check the logs and see this issue :

{"at":"2020-08-18T20:10:06.96Z","env":"1.18.0:9e06d","ns":["cardano.node.DnsSubscription"],"data":{"event":"Connection Attempt Start, destination 54.151.49.138:3001","kind":"SubscriptionTrace","domain":"\"relays-new.cardano-testnet.iohkdev.io\""},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Notice","thread":"815"} {"at":"2020-08-18T20:10:06.98Z","env":"1.18.0:9e06d","ns":["cardano.node.DnsSubscription"],"data":{"event":"Connection Attempt Start, destination 18.132.238.21:3001","kind":"SubscriptionTrace","domain":"\"relays-new.cardano-testnet.iohkdev.io\""},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Notice","thread":"816"} {"at":"2020-08-18T20:10:07.00Z","env":"1.18.0:9e06d","ns":["cardano.node.DnsSubscription"],"data":{"event":"Connection Attempt End, destination 18.132.238.21:3001 outcome: ConnectSuccessLast","kind":"SubscriptionTrace","domain":"\"relays-new.cardano-testnet.iohkdev.io\""},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Notice","thread":"816"} {"at":"2020-08-18T20:10:07.00Z","env":"1.18.0:9e06d","ns":["cardano.node.Handshake"],"data":{"event":"Send MsgProposeVersions (fromList [(NodeToNodeV_1,TInt 1097911063),(NodeToNodeV_2,TInt 1097911063),(NodeToNodeV_3,TInt 1097911063)])","kind":"HandshakeTrace","bearer":"ConnectionId {localAddress = 161.35.199.183:33215, remoteAddress = 18.132.238.21:3001}"},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Info","thread":"816"} {"at":"2020-08-18T20:10:07.00Z","env":"1.18.0:9e06d","ns":["cardano.node.ErrorPolicy"],"data":{"event":"ErrorPolicySuspendConsumer (Just (ConnectionExceptionTrace (SubscriberError {seType = SubscriberParallelConnectionCancelled, seMessage = \"Parallel connection cancelled\", seStack = []}))) 1s","kind":"ErrorPolicyTrace","address":"54.151.49.138:3001"},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Notice","thread":"64"} {"at":"2020-08-18T20:10:07.01Z","env":"1.18.0:9e06d","ns":["cardano.node.Handshake"],"data":{"event":"Recv MsgAcceptVersion NodeToNodeV_2 (TInt 1097911063)","kind":"HandshakeTrace","bearer":"ConnectionId {localAddress = 161.35.199.183:33215, remoteAddress = 18.132.238.21:3001}"},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Info","thread":"816"} {"at":"2020-08-18T20:10:07.02Z","env":"1.18.0:9e06d","ns":["cardano.node.ChainSyncClient"],"data":{"kind":"ChainSyncClientEvent.TraceFoundIntersection"},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Info","thread":"829"} {"at":"2020-08-18T20:10:07.04Z","env":"1.18.0:9e06d","ns":["cardano.node.ChainSyncClient"],"data":{"kind":"ChainSyncClientEvent.TraceRolledBack","tip":{"kind":"BlockPoint","slot":21365,"headerHash":"d4e97a6bc69d123ea649d70b3978f8bf715fe6f5260957320f7c7e73f9b49b79"}},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Notice","thread":"829"} {"at":"2020-08-18T20:10:07.04Z","env":"1.18.0:9e06d","ns":["cardano.node.ChainSyncClient"],"data":{"kind":"ChainSyncClientEvent.TraceDownloadedHeader","block":{"kind":"BlockPoint","slot":21366,"headerHash":"1b7031d140af37c0eed1f4cc56581c63565567532f4b0b60dbe460c3231bf9e3"}},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Info","thread":"829"} {"at":"2020-08-18T20:10:07.04Z","env":"1.18.0:9e06d","ns":["cardano.node.ChainSyncClient"],"data":{"exception":"InvalidBlock (At (Block {blockPointSlot = SlotNo 21366, blockPointHash = 1b7031d140af37c0eed1f4cc56581c63565567532f4b0b60dbe460c3231bf9e3})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationUTxOValidationError (UTxOValidationUTxOError (UTxOMissingInput (TxInUtxo 0d712f00a600aedb5b725a5e4f391dad44fa7f319655e38629514033ca8df0cc 0)))}))))","kind":"ChainSyncClientEvent.TraceException"},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Warning","thread":"829"} {"at":"2020-08-18T20:10:07.04Z","env":"1.18.0:9e06d","ns":["cardano.node.DnsSubscription"],"data":{"event":"Application Exception: 18.132.238.21:3001 InvalidBlock (At (Block {blockPointSlot = SlotNo 21366, blockPointHash = 1b7031d140af37c0eed1f4cc56581c63565567532f4b0b60dbe460c3231bf9e3})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationUTxOValidationError (UTxOValidationUTxOError (UTxOMissingInput (TxInUtxo 0d712f00a600aedb5b725a5e4f391dad44fa7f319655e38629514033ca8df0cc 0)))}))))","kind":"SubscriptionTrace","domain":"\"relays-new.cardano-testnet.iohkdev.io\""},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Error","thread":"816"} {"at":"2020-08-18T20:10:07.04Z","env":"1.18.0:9e06d","ns":["cardano.node.ErrorPolicy"],"data":{"event":"ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (InvalidBlock (At (Block {blockPointSlot = SlotNo 21366, blockPointHash = 1b7031d140af37c0eed1f4cc56581c63565567532f4b0b60dbe460c3231bf9e3})) (ValidationError (ExtValidationErrorLedger (HardForkLedgerErrorFromEra Z (WrapLedgerErr {unwrapLedgerErr = ChainValidationUTxOValidationError (UTxOValidationUTxOError (UTxOMissingInput (TxInUtxo 0d712f00a600aedb5b725a5e4f391dad44fa7f319655e38629514033ca8df0cc 0)))}))))))) 200s 200s","kind":"ErrorPolicyTrace","address":"18.132.238.21:3001"},"app":[],"msg":"","pid":"19597","loc":null,"host":"ubuntu-s","sev":"Warning","thread":"64"}

mainnet-topology.json

{
"Producers": [
{
"addr": "relays-new.cardano-mainnet.iohk.io",
"port": 3001,
"valency": 2
}
]
}
tanzeelrana commented 4 years ago

never mind my config files were wrong