[Bug] Potential temporary sync stall from weak-link or malicious peers

🐛 Bug Report

A circumstance where block requests are forced to timeout can be reached when a peer with a pending request does not respond with a block response. This is similar to the temporary stall outlined in #3321.

Steps to Reproduce

prevent a block response from being created from the validator or the client's router
observe syncing nodes using the modified nodes as peers get stuck for ~10 minutes when reaching a block with an "incomplete request"

We encountered a 10 minute stall during a sync of a node on high performance hardware after merging the latest client sync fixes and deduced it may be related to a swarm of low-quality clients as connected peers (from another test). We were able to resume block sync immediately by forcibly disconnecting the weak links.

The 34.16.96.117 IP is one of 20 16core servers under extreme stress from 10 clients syncing blocks with dozens of transactions per block and as a result is unreliably able to respond to block requests, preventing a block request from being marked as complete.

Expected Behavior

Blocks to sync without stalls when a peer unable to respond.

Potential Fixes

timeout for block requests on a peer-node basis
disconnect from peers that are unreliable
allow for a certain # of peers to be unresponsive when marking a request as complete in block_sync.rs
shorter total block request timeouts

Your Environment

snarkOS version: mainnet-staging on canary + rwlock (#3321) + node ip fixes (#3318)
Rust version: 1.76
OS: Ubuntu 24.04

AleoNet / snarkOS