search

near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.3k stars 603 forks source link

Blocks may be never finalised on network start #10558

Open Longarithm opened 5 months ago

Longarithm commented 5 months ago

On StatelessNet start @khorolets discovered that there are no blocks on chain: https://near.zulipchat.com/#narrow/stream/407237-pagoda.2Fcore.2Fstateless-validation/topic/block.20finality.20issues/near/419233634

The issue persisted for ~3 hours and got autoresolved then, and we don't know why. It was too early for epoch transition. This doesn't look relevant only to the protocol change introduced by stateless validation. Debug logs show that no block endorsements were being sent, only the skips, which led to skipping absolutely all blocks on even heights.

Also this was reproduced on localnet with 1-2 nodes. However, adversenet testing didn't show that.

I don't think it will happen again for statelessnet as more nodes will improve network quality, but it could be nice to understand what happens inside Doomslug::get_timer and the reason behind the issue.

Debug branch https://github.com/near/nearcore/commit/ab20f8c28d97ded299b9362bbb3f2389443f0425 2.6 GB logs https://drive.google.com/file/d/1pYc9pZcyCAcAUjloYRzxRwkIFhJEyRPp/view?usp=drive_link

andrei-near commented 3 weeks ago

We have this situation in Statelessnet near_final_block_height 119519718 near_block_height_head 119525916