search

near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.34k stars 632 forks source link

Blocks may be never finalised on network start #10558

Open Longarithm opened 10 months ago

Longarithm commented 10 months ago

On StatelessNet start @khorolets discovered that there are no blocks on chain: https://near.zulipchat.com/#narrow/stream/407237-pagoda.2Fcore.2Fstateless-validation/topic/block.20finality.20issues/near/419233634

The issue persisted for ~3 hours and got autoresolved then, and we don't know why. It was too early for epoch transition. This doesn't look relevant only to the protocol change introduced by stateless validation. Debug logs show that no block endorsements were being sent, only the skips, which led to skipping absolutely all blocks on even heights.

Also this was reproduced on localnet with 1-2 nodes. However, adversenet testing didn't show that.

I don't think it will happen again for statelessnet as more nodes will improve network quality, but it could be nice to understand what happens inside Doomslug::get_timer and the reason behind the issue.

Debug branch https://github.com/near/nearcore/commit/ab20f8c28d97ded299b9362bbb3f2389443f0425 2.6 GB logs https://drive.google.com/file/d/1pYc9pZcyCAcAUjloYRzxRwkIFhJEyRPp/view?usp=drive_link

andrei-near commented 5 months ago

We have this situation in Statelessnet near_final_block_height 119519718 near_block_height_head 119525916