search

KiraCore / testnet

public testnet coordination
Other
31 stars 28 forks source link

A lot of validators missing blocks #11

Closed Creamers158 closed 3 years ago

Creamers158 commented 3 years ago

My node ran for more that 10hours without missing any blocks. Now it seems to have bit streaks missing signing blocks.

2021-04-04T11:31:15.576875220Z  ^[[90m11:31AM^[[0m ^[[32mINF^[[0m enterPrecommit(24200/0). Current: 24200/0/RoundStepPropose ^[[36mheight=^[[0m24200 ^[[36mmodule=^[[0mconsensus ^>

2021-04-04T11:31:15.577053027Z  ^[[90m11:31AM^[[0m ^[[32mINF^[[0m enterPrecommit: No +2/3 prevotes during enterPrecommit. Precommitting nil. ^[[36mheight=^[[0m24200 ^[[36mmodule=>

2021-04-04T11:31:15.580595436Z  ^[[90m11:31AM^[[0m ^[[32mINF^[[0m Signed and pushed vote ^[[36merr=^[[0mnull ^[[36mheight=^[[0m24200 ^[[36mmodule=^[[0mconsensus ^[[36mround=^[[0m>

2021-04-04T11:31:15.580612488Z  ^[[90m11:31AM^[[0m ^[[32mINF^[[0m enterCommit(24200/0). Current: 24200/0/RoundStepPrecommit ^[[36mcommitRound=^[[0m0 ^[[36mheight=^[[0m24200 ^[[36>

2021-04-04T11:31:15.580615724Z  ^[[90m11:31AM^[[0m ^[[32mINF^[[0m Commit is for a block we don't know about. Set ProposalBlock=nil ^[[36mcommit=^[[0m81D802FDA4A86C2D87275DB687D45>

2021-04-04T11:31:15.580625122Z  ^[[90m11:31AM^[[0m ^[[32mINF^[[0m Attempt to finalize failed. We don't have the commit block. ^[[36mcommit-block=^[[0m81D802FDA4A86C

At some point it looks like the cpu intensive manager lacks behind or the consensus with a lot of validators having kind alike problems have isues.

Creamers158 commented 3 years ago

I've restarted ALL Dockers using the kira manager. Seemed to solve the issue so far (so good)

Creamers158 commented 3 years ago

Some additional findings of the "block we don't know about" https://github.com/tendermint/tendermint/issues/3721 https://github.com/tendermint/tendermint/issues/6157

Or in short someone said: I should mention that one of the oldest issues in Tendermint is that once a nodes moves into consensus, even if that node starts lagging behind that it will never go back into fast sync (#129) It's something we hope to tackle in the next two major releases.

Creamers158 commented 3 years ago

Also when missing blocks I see differences between sentry height and validator height block image

Creamers158 commented 3 years ago

Okay, after restarting the containers it didnt seem to help. Got inactive state and tried to activate again but then I got this: image

Creamers158 commented 3 years ago

after submitting several times it became active again but not signing blocks.

I decided to reboot the whole server and now it is signing again.

asmodat commented 3 years ago

thank you, resolved on branch testnet-2 if mahcine have min required specs