It's possible for a provider to not be "stalled" as its block number is still increasing, but to just be way behind the actual tip. This happened to us with a base provider https://rpc.notadegen.com/base - its latest block was 12 hrs old and slowly increasing.
Solution
A provider should be deemed unhealthy if its latest block is far behind the rest of the pack. We'll want some tolerance for normal race conditions, I'm unsure exactly what tbh, but iirc rollups with crazy fast block times can sometimes be at "tip" but disagree on what that tip is by a decent amount
Problem
It's possible for a provider to not be "stalled" as its block number is still increasing, but to just be way behind the actual tip. This happened to us with a base provider https://rpc.notadegen.com/base - its latest block was 12 hrs old and slowly increasing.
Solution
A provider should be deemed unhealthy if its latest block is far behind the rest of the pack. We'll want some tolerance for normal race conditions, I'm unsure exactly what tbh, but iirc rollups with crazy fast block times can sometimes be at "tip" but disagree on what that tip is by a decent amount