[Bug] 85400 CanaryNet Network Stall

[damons]

🐛 Bug Report

All nodes were stuck at 85400, except for Demox which was stuck at 85398 and ANF node 4, which was stuck at 84348 due to a crash caused by reasons unknown (potentially high number of IO intensive block requests to the API). High API latency observed, up to almost a minute.

Initial Findings:

• Lots of unconfirmed transmissions in bursts.
• Recovery requires all nodes to shutdown (or at least a sufficient number of nodes including exclusion of the nodes that haven’t been restarted yet)
• Nodes can advance without quorum
• Restarting a node can cause a ledger to jump back dozens or hundreds of blocks

Potential Optimizations:

• Are we checking transactions as existing in the ledger before we gossip them?
• Can block specific ips or should we fallback on firewalls?
• Ledger should support get ranges

Steps to Reproduce

Currently investigating. This issue is to track progress.

Expected Behavior

Network continues and block production progresses.

Your Environment

[vicsn]

All nodes were stuck at 85400, except for Demox which was stuck at 85398

This looks very similar to what Devnet experienced on 04/11/2024. The resolution for this was basically: https://github.com/AleoHQ/snarkOS/pull/3211

Without further data, I suggest we close this issue.