search

input-output-hk / cardano-sl

Cryptographic currency implementing Ouroboros PoS protocol
Apache License 2.0
3.77k stars 630 forks source link

Failed to verify blocks: stake distribution for epoch #179 is unknown #4277

Closed shuhuan2014 closed 4 years ago

shuhuan2014 commented 4 years ago

Can not synchronize the epoch ##179 ,

anyone knows how to solve this problem ?

[cardano-sl.*production*:Warning:ThreadId 353] [2020-03-12 03:32:56.35 UTC] handleRecoveryE: error handling nodeId="13.228.101.136:3000:0", header=39a8065bdcfbe6dd: BlockNetLogicException: DialogUnexpected "s**_take distribution for epoch #179 is unknown_**"
[cardano-sl.*production*:Warning:ThreadId 353] [2020-03-12 03:32:56.35 UTC] Recovery mode communication dropped with peer "13.228.101.136:3000:0"
[cardano-sl.*production*:Debug:ThreadId 353] [2020-03-12 03:32:56.35 UTC] Attempting to restart recovery
[cardano-sl.*production*:Debug:ThreadId 353] [2020-03-12 03:32:58.35 UTC] Recovery triggered, requesting tips from neighbors
[cardano-sl.diffusion.outboundqueue.self:Debug:ThreadId 353] [2020-03-12 03:32:58.35 UTC] inFlight = fromList []
[cardano-sl.diffusion.outboundqueue.self:Debug:ThreadId 353] [2020-03-12 03:32:58.35 UTC] Rejected alternative NodeId 13.228.101.136:3000:0 as it has a recent failure
[cardano-sl.diffusion.outboundqueue.self:Warning:ThreadId 353] [2020-03-12 03:32:58.35 UTC] could not choose suitable alternative from [NodeId 13.228.101.136:3000:0]
[cardano-sl.diffusion.outboundqueue.self:Error:ThreadId 353] [2020-03-12 03:32:58.35 UTC] enqueue instruction EnqueueAll NodeRelay (MaxAhead 1) PHigh failed to enqueue message MsgRequestBlockHeaders Nothing to forwarding sets [[NodeId 13.228.101.136:3000:0]]
[cardano-sl.*production*:Debug:ThreadId 353] [2020-03-12 03:32:58.35 UTC] Finished requesting tips for recovery
erikd commented 4 years ago

What version is this? Is it a release version or something compiled from Git? If the latter, please supply the git hash.

shuhuan2014 commented 4 years ago

What version is this? Is it a release version or something compiled from Git? If the latter, please supply the git hash.

commit 1a792d7cd0f0c93a0f0c28f66372bce3c3808dbd (HEAD -> master, tag: 3.1.0, origin/master) Merge: 6e413d2d4 263b8cdc5 Author: iohk-bors[bot] 43231472+iohk-bors[bot]@users.noreply.github.com Date: Tue Sep 24 13:39:43 2019 +0000

shuhuan2014 commented 4 years ago

[cardano-sl.diffusion.outboundqueue.self:Warning:ThreadId 2120] [2020-03-12 04:00:56.44 UTC] sending MsgRequestBlocks (fromList [NodeId 13.228.101.136:3000:0]) to NodeId 13.228.101.136:3000:0 failed with BlockNetLogicException: DialogUnexpected "stake distribution for epoch #179 is unknown" :: SomeException [cardano-sl.production:Warning:ThreadId 353] [2020-03-12 04:00:56.44 UTC] handleRecoveryE: error handling nodeId="13.228.101.136:3000:0", header=296359d690c79707: BlockNetLogicException: DialogUnexpected "stake distribution for epoch #179 is unknown" [cardano-sl.production:Warning:ThreadId 353] [2020-03-12 04:00:56.44 UTC] Recovery mode communication dropped with peer "13.228.101.136:3000:0" [cardano-sl.production:Debug:ThreadId 353] [2020-03-12 04:00:56.44 UTC] Attempting to restart recovery

shuhuan2014 commented 4 years ago

[cardano-sl.production:Debug:ThreadId 2854] [2020-03-12 04:14:16.86 UTC] Rolling: verifying [cardano-sl.production:Info:ThreadId 2854] [2020-03-12 04:14:16.86 UTC] slogVerifyBlocks: Consensus era is OBFT ObftLenient [cardano-sl.diffusion:Debug:ThreadId 2853] [2020-03-12 04:14:17.05 UTC] Updating Window: 1024 to 2048 [cardano-sl.diffusion:Debug:ThreadId 2853] [2020-03-12 04:14:17.17 UTC] Updating Window: 1024 to 2048 [cardano-sl.production:Debug:ThreadId 2854] [2020-03-12 04:14:17.96 UTC] usVerifyBlocks OBFT: We're on epoch boundary. Running processGenesisBlock [cardano-sl.production:Debug:ThreadId 2854] [2020-03-12 04:14:17.96 UTC] Rolling: run applyAMAP: apply as much as possible after verifyBlocksPrefix failure: ApplyBlocksVerifyFailure (VerifyBlocksError "stake distribution for epoch #179 is unknown")

erikd commented 4 years ago

I would suggest trying tag https://github.com/input-output-hk/cardano-sl/releases/tag/3.2.0 .

shuhuan2014 commented 4 years ago

I would suggest trying tag https://github.com/input-output-hk/cardano-sl/releases/tag/3.2.0 .

should i rebuild again ?

erikd commented 4 years ago

You can only try that tag by rebuilding.

shuhuan2014 commented 4 years ago

You can only try that tag by rebuilding.

I have many wallets created in the old version.

how to transfer to new version?

erikd commented 4 years ago

No need to transfer. Tag 3.2.0 or cardano-sl is compatible with tag 3.1.0.

shuhuan2014 commented 4 years ago

No need to transfer. Tag 3.2.0 or cardano-sl is compatible with tag 3.1.0.

Do i need to delete the old folder: state-wallet-mainnet ?

erikd commented 4 years ago

Have you tried deleting that folder and re-running with tag 3.1.0? If you haven't that would be worth trying first.

shuhuan2014 commented 4 years ago

Have you tried deleting that folder and re-running with tag 3.1.0? If you haven't that would be worth trying first.

OK, i will try to change the folder name to make backup

shuhuan2014 commented 4 years ago

Now start to scratch.

BUT the problem is the API can not get data of my ADA wallet. should i wait ?

curl -X GET https://127.0.0.1:8090/api/v1/wallets \

-H "Accept: application/json; charset=utf-8" \ -H "Content-Type: application/json; charset=utf-8" \ --cacert /root/cardano/cardano-sl/state-wallet-mainnet/tls/client/ca.crt \ --cert /root/cardano/cardano-sl/state-wallet-mainnet/tls/client/client.pem \

{"data":[],"status":"success","meta":{"pagination":{"totalPages":1,"page":1,"perPage":10,"totalEntries":0}}}

[cardano-sl.production:Info:377] [2020-03-12 04:45:18.96 UTC] Trying to apply blocks w/o rollback. First 3: [MainBlockHeader: hash: 91237e7e86349961cb23f5f1486231fa1ea127ed6531f20717443d73a6a93bca previous block: 3616fdace43c820151fb83c7919f792ccb10cf8c801c11dfa1c0129ee139e6b9 slot: 4224th slot of 0th epoch difficulty: 4225 leader: pub:993a8f05 signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:993a8f05, dPk = pub:89c29f8c } } block: v0.0.0 software: cardano-sl:0

erikd commented 4 years ago

You have just deleted the state directory. You have to wait for the node to sync.

shuhuan2014 commented 4 years ago

You have just deleted the state wallet. You have to wait for the node to sync.

the sync process stoped automatically.

anything wrong ?

[cardano-sl.*production*:Debug:383] [2020-03-12 04:53:47.94 UTC] Rolling: verifying
[cardano-sl.*production*:Debug:383] [2020-03-12 04:53:47.94 UTC] verifyBlocksPrefix: 64
[cardano-sl.*production*:Info:383] [2020-03-12 04:53:47.94 UTC] slogVerifyBlocks: Consensus era is Original
[cardano-sl.*production*:Debug:383] [2020-03-12 04:53:47.98 UTC] Rolling: Verification done, applying unsafe block
[cardano-sl.node:Debug:316] [2020-03-12 04:53:47.99 UTC] applying some blocks (non-rollback)
[cardano-sl.*production*:Info:383] [2020-03-12 04:53:48.00 UTC] Verifying and applying blocks done
[cardano-sl.*production*:Debug:383] [2020-03-12 04:53:48.00 UTC] MemPool metrics release: ApplyBlock modify time was 62231mcs size is 0
[cardano-sl.*production*:Debug:383] [2020-03-12 04:53:48.00 UTC] Not relaying block in recovery mode
erikd commented 4 years ago

How do you know it stopped?

shuhuan2014 commented 4 years ago

the process of cardano-node is dead. and the port 8090 can not access

erikd commented 4 years ago

If it died where is the error message?

shuhuan2014 commented 4 years ago

this is the last log message

[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.45 UTC] Rolling: Verification done, applying unsafe block
[cardano-sl.node:Debug:316] [2020-03-12 05:22:34.46 UTC] applying some blocks (non-rollback)
[cardano-sl.*production*:Info:378] [2020-03-12 05:22:34.48 UTC] Verifying and applying blocks done
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] MemPool metrics release: ApplyBlock modify time was 84697mcs size is 0
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] Not relaying block in recovery mode
[cardano-sl.*production*:Info:378] [2020-03-12 05:22:34.48 UTC] Blocks have been adopted: [38b0c38769dd2b1e, be3a2725f4695fca, d31d7d44634d64ff, b36ec10eb10d63d7, 4d2303739126f84a, 6baa4ce3b11c1a7f, e8358531e2e28e07, eafb3f7257ee38f1, 6dc3ded31a123b67, fa44be6e1f88ecd7, e4f51a40d1b54b31, 354f2dac9e4dec3b, 3ac3928af88b54a8, 3c6fb54a8fd67033, 61127b944bee18f5, 8368cf64d627a08f, 1d291ce452cdc5d9, ae973bd56614bdce, 08beb19507a57b18, 75b557e264306925, 369104b94f8016e0, 5fb5d4b7f88362b0, 3bcfe877818acf94, ddb142c400416b4c, e74fe942c2c17f69, 9cae637d36a87351, 13c309cdf0477b3a, 60bdcbf68417948f, 52ec5b5c11b536c5, 7e399636b729d7e8, 30f39dfffb1f399b, 2ec76ae576112c7b, 957e3b1927a8f2aa, ecb55f8ee182e2ae, 322ea62c51189cfa, 0f5b72e01c036dd9, 0bad7da8416b3245, 6fc4c39cd9034784, 8a7341fc92d67944, 92e142bf4b897f59, fd1bdc49906a8659, 4f60d3dc26498f8b, 015f742cb8918759, e951c51561f001a9, 8712f6ea5246dfac, 6c919971853f4f1b, f804bc918dba0606, fc8260a9b2ccda2b, ff572f771ea84942, 3456a3455f384d1c, ca285d3eede34c99, 120915041dcc809b, 32d1a74e5173e837, 01b4177c4a36c504, eb383dbb43fd8753, e8a3c05b4798c440, 0ecffde407d01861, 2ec27053968127e6, 750bb1f525703a20, 1e1129ee1061401b, c088e3c8c61fc3d8, 65f7c64e1271d2c1, 8ff2b571d097a51d, dbe5a93c669a174d]
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] handleBlocks: processing
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] Handling block w/ LCA, which is dbe5a93c
[cardano-sl.*production*:Info:378] [2020-03-12 05:22:34.48 UTC] Trying to apply blocks w/o rollback. First 3: [MainBlockHeader:
    hash: 0a9d610dcfd81aa6c5728ed08ca7e6d80c8f4a579a42cc9642b118d10254fa3e
    previous block: dbe5a93c669a174d6fc92ce6d84212d313f7649df7f3fa6ae0d758d069737a2d
    slot: 15269th slot of 20th epoch
    difficulty: 447213
    leader: pub:9a6fa343
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:9a6fa343, dPk = pub:8b532076 } }
    block: v0.1.0
    software: cardano-sl:0
, MainBlockHeader:
    hash: 9d49740a184042b0014d41234da53520d2a9785e1153fc7b0b9f6a2dd9c68201
    previous block: 0a9d610dcfd81aa6c5728ed08ca7e6d80c8f4a579a42cc9642b118d10254fa3e
    slot: 15270th slot of 20th epoch
    difficulty: 447214
    leader: pub:50733161
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:50733161, dPk = pub:e8c03a03 } }
    block: v0.1.0
    software: cardano-sl:0
, MainBlockHeader:
    hash: c0a8f12a7062aee7daca2be1570bdb7626f646266960d8ef0e9f298c80f0fd3b
    previous block: 9d49740a184042b0014d41234da53520d2a9785e1153fc7b0b9f6a2dd9c68201
    slot: 15271st slot of 20th epoch
    difficulty: 447215
    leader: pub:d2965c86
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:d2965c86, dPk = pub:9180d818 } }
    block: v0.1.0
    software: cardano-sl:0
]
Last 3: [MainBlockHeader:
    hash: 78146314179c040493cc285216487cfc9f31cac71221a98293547e924a540e1f
    previous block: f769f3a8e5d7482c271bcb06cecd208b23d954a5b74a2b0f99b2d94066164ae1
    slot: 15330th slot of 20th epoch
    difficulty: 447274
    leader: pub:d2965c86
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:d2965c86, dPk = pub:9180d818 } }
    block: v0.1.0
    software: cardano-sl:0
, MainBlockHeader:
    hash: 2226aafb6f4713e8580207b32179dd2ccb616575577925bc542c02ad10efd595
    previous block: 78146314179c040493cc285216487cfc9f31cac71221a98293547e924a540e1f
    slot: 15331st slot of 20th epoch
    difficulty: 447275
    leader: pub:1bc97a2f
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:1bc97a2f, dPk = pub:61261a95 } }
    block: v0.1.0
    software: cardano-sl:0
, MainBlockHeader:
    hash: efcb0e8d4f373aceab733b1a57b8e163a469a922996c45f62fd9126e7151e85b
    previous block: 2226aafb6f4713e8580207b32179dd2ccb616575577925bc542c02ad10efd595
    slot: 15332nd slot of 20th epoch
    difficulty: 447276
    leader: pub:26566e86
    signature: BlockPSignatureHeavy: Proxy signature { psk = ProxySk { w = #0, iPk = pub:26566e86, dPk = pub:f14f712d } }
    block: v0.1.0
    software: cardano-sl:0
]
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] MemPool metrics wait: ApplyBlock queue length is 1
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] MemPool metrics acquire: ApplyBlock wait time was 10mcs
[cardano-sl.*production*:Info:378] [2020-03-12 05:22:34.48 UTC] Verifying and applying blocks...
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] Rolling: verifying
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.48 UTC] verifyBlocksPrefix: 64
[cardano-sl.*production*:Info:378] [2020-03-12 05:22:34.48 UTC] slogVerifyBlocks: Consensus era is Original
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.54 UTC] Rolling: Verification done, applying unsafe block
[cardano-sl.node:Debug:316] [2020-03-12 05:22:34.55 UTC] applying some blocks (non-rollback)
[cardano-sl.*production*:Info:378] [2020-03-12 05:22:34.57 UTC] Verifying and applying blocks done
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.57 UTC] MemPool metrics release: ApplyBlock modify time was 86828mcs size is 0
[cardano-sl.*production*:Debug:378] [2020-03-12 05:22:34.57 UTC] Not relaying block in recovery mode
erikd commented 4 years ago

There is nothing in that log indicating that the program terminated.

shuhuan2014 commented 4 years ago

the PID of cardano-node is dead. and the port 8090 can not access

shuhuan2014 commented 4 years ago

and no log generate again, stop there

erikd commented 4 years ago

What about the kernel log? Anything there?

erikd commented 4 years ago

The kernel logs might indicate two things:

shuhuan2014 commented 4 years ago

The kernel logs might indicate two things:

  • the process being killed by the OOM killer (so add memory)
  • the process segfaulting (which is probably also a consequence of insufficient memory)

Thank you, i will change to another Linux server to try

erikd commented 4 years ago

Did you see either of these things in the linux kernel log? If you have not seen either of those two kernel messages, that may not be the problem.

erikd commented 4 years ago

Any update on this?

erikd commented 4 years ago

Closing this.