paradox-tt
commented
1 week ago What's also a bit odd is that the slash was reported for era 7126 which is 6 eras (36hrs ago) but the slash report was made approximately 10hrs ago
Open paradox-tt opened 1 week ago
paradox-tt
commented
1 week ago What's also a bit odd is that the slash was reported for era 7126 which is 6 eras (36hrs ago) but the slash report was made approximately 10hrs ago
stakeworld
commented
6 days ago Also odd that you get a slash while in a sync state, so not yet actively validating blocks. Almost as if the finality worker was using old info (from when the node went down, some kind of buffer?) and tried something with that old info against the already finalized chain? Hopefully someone with more expertise in this area can take a look.
bkchr
commented
4 days ago Sep 19 18:53:17 Germany15 polkadot[995]: 2024-09-19 18:53:17 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (24975072, 0xa17977c835990ef96666c89cb29f224332b99e22745ba87d510fe1772f62fdf9), block_descriptions: [BlockDescription { block_hash: 0x217e7dbbd66bcae7f8d426250e0aab48fc0a46aa5027c1a90b6c22fef9d76f42, session: 42142, candidates: [0x0cb279f0237515b9b9202be17571a16e75d199f955df016bad2ad38fe933805f, 0x1a05a97503fc132be044352fac8b2b260ff481a094eee3ca94493fc0cc261283, 0xf8aab0c04e23eeb042228a4fc6ba1d6a1c80701d4daf82be225ff2e1db02e160, 0xfbaf02fc2460236c0d2cb32c14ae1ae5c3713b558fe6e8daf556652d97a0d546, 0x497bb3575e0c653a16c40a41a599552a1b5ae4c88243ecad99672aca920f5eec, 0xdb002b2b7ba57e25d56a9a9b5e663be50655f156567eb71ed9799411538275c2, 0x0d46374fbac208504f3e57f82de3510dd12b48c555153d1bdc4c3cbae62da309, 0x39a27e4c340995a8d0e8a55c31e76d09bfc9f65a48ee6fbd5269049ea2272174, 0x3ca409ba5612df6bc917d6f084421ea044282da6d04ea864c85f2f615c13286c, 0x10058a193451c6138356b818c32192c93831943004f9b0bc86d39f886de4a00c, 0xb1bea09dddc1ece478db99c6e4f4635f291b6886784a182e46045011af856b9f, 0xae446b5213ba2a4b239c33eed6a06bee59bbbca260be9122fbd93354504dc8dc, 0x3a7d8a7f022f40d7db9efdfa6ec139bc75a255b7dd5da5b4132a3c1a70091fe3, 0xabc05e30877ee527c9f767c1d000fd6db145633c92ee5bfbac46e8d789c4e7e4, 0xf8427ed25e66976ca0381cd54e5e346c502af8ecb08aadaac774b52242c141b0, 0xbf5cb955545e51a061a1c32933c7fb0d4a0d2df11f098910d5357ef6d879081f, 0xdace540d6e2a67d58e7ac0123a3c307d0d6a65aafd460f98fcfde9244a10074a, 0x0c8bdcc7d8dc7862bbf4a6eeb574f58d9d9d9325e5d5a2adbd6cfbcf9ecfc9e1, 0xbb1b926c7eb45110cf1cf345de809498a2c19aceb96238a38f24d47b20aecf2c, 0x9537c5cced83718dea4e6b64feba200dfe741ab699c3c20eb41b0c82f94219b4, 0x712a5ad1822c4de1b6b2a8cd0fc03b7aaecefb340da81e3df428e8de57ebe430, 0x0265fed98905630d89859baad0b56d4a6963f093d8b47dbe172567dc54d1011a] }], tx: Sender { complete: false } }
Sep 19 18:53:17 Germany15 polkadot[995]: 2024-09-19 18:53:17 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled)
Sep 19 18:53:17 Germany15 polkadot[995]: 2024-09-19 18:53:17 Detected prevote equivocation in the finality worker: Equivocation { round_number: 3381, identity: Public(f2178a9c2a562918b17dc069c468467acbb7a0480cf9bb6461f844ce79e5447d (J3k6honQ...)), first: (Prevote { target_hash: 0x217e7dbbd66bcae7f8d426250e0aab48fc0a46aa5027c1a90b6c22fef9d76f42, target_number: 24975073 }, Signature(ff66607cc73600dc22c55982c5062a11fc1e4b2327cd5ee7c90292ab0f264a14be8298d7778bc9274067178f1c2d739cf52c9c82471564bc0df6f6c1d0bf1705)), second: (Prevote { target_hash: 0xa17977c835990ef96666c89cb29f224332b99e22745ba87d510fe1772f62fdf9, target_number: 24975072 }, Signature(675dcdc5277ca44c17f58ff612bd887014a501fb998f5f4af2ba4a271c94ee967e6a7be1257287c054f2372d00c7570dd383cb8791fc99721e5e21a6cb908a04)) }
Sep 19 18:53:17 Germany15 polkadot[995]: 2024-09-19 18:53:17 Error reporting prevote equivocation: safety invariant has been violated: Refraining from sending equivocation report for our own equivocation.Looking at the code it seems we have tried to prevote again on the base_blockr, which is probably the last finalized block on which we already voted. IMO this seems to indicate some bug, but I will try to confirm.
paradox-tt
commented
4 days ago Thanks Basti, I'll register a referendum to cancel the slash even if the vote has no on-chain consequence.
andresilva
commented
3 days ago @paradox-tt It's not evident from the logs but how was the node shut down? Was it a clean shut down? The node persists the block it voted for in order to prevent equivocating in case it's restarted, but if the node isn't shut down properly (e.g. the host machine is powered off), it could be that this isn't properly persisted.
bkchr
commented
3 days ago Sep 18 16:22:01 Germany15 polkadot[1692015]: 2024-09-18 16:22:01 💤 Idle (32 peers), best: #24975077 (0x22e9…a325), finalized #24975073 (0x217e…6f42), ⬇ 939.4kiB/s ⬆ 930.3kiB/s -- Boot fd882bac0965476ab0d377d81e143aaa --
The logs seem to indicate that it wasn't a clean shutdown. If @paradox-tt can confirm this, the vote was probably not persisted.
paradox-tt
commented
3 days ago @andresilva I can confirm the shutdown was NOT clean.
Just asking, when the node comes back up and is not is syncing should a vote/prevote still be sent?
andresilva
commented
3 days ago On a high-level I would say no, we could detect whether we are major syncing or not and then decide whether the grandpa voter does anything based on that. The reason we are not doing that is that finality and block production aren't tied to each other, so you could be 100 blocks behind while syncing and your vote for finality would still be useful (e.g. if finality was also lagging 100 blocks). It also makes the code simpler since there's less edge cases to think about. And waiting for sync to be finished wouldn't necessarily guarantee that we wouldn't equivocate anyway once it finishes.
I think the root issue here is that the information about what the validator voted on wasn't persisted. This can happen because of a dirty shutdown, but we can also improve this situation. Currently it seems that there is no way to explicitly sync the database (and thus make sure all data is flushed to disk), if we add support for this then the situation should be resolved, because only after being sure that the vote information is actually persisted on disk, do we push the vote to the network. Assuming this exists if the node crashes then we are sure that: either the node didn't vote OR if it voted and the vote reached the network then the data was definitely persisted on disk, and therefore upon restarting it will not vote for something else.
(Off-topic: FWIW I would personally vote in favor of reverting the slash.)
Hey guys,
I endured some downtime on one of my Kusama nodes, when it restarted it received a slash due to equivocation with details below.
The slashed amount is fairly low but I would still like to have the technicalities looked at to see if this is a bug or user error. If it's a bug I would then seek to have the slash revoked and if not I'll adjust my proceedures to prevent reoccurence.