[pool/builder] Reorgs from inconsistent RPCs lead to lost UOs

[dancoombs]

Describe the bug An issue we have when running on top of inconsistent RPC providers during reorgs:

1. Builder creates a bundle and sends a validation request to to the RPC provider which routes it to a node which is currently on block N*
2. In N* a UO landed, so the RPC request returns with invalid nonce, and we remove the UO from the mempool
3. N is eventually reorged, however out pool chain tracking logic never saw N because we're using an inconsistent provider and it was routed directly to N where UO was never mined

In this case, we've removed the UO from the mempool, but it was never part of the "canonical chain" and didn't land.

I think we need to give the pool full control over UO removal due to nonce increases which may avoid this. Unsure if there are any further impacts though

To reproduce Not easy, but we've seen it happen.

Expected behavior N* shouldn't be lost.

Logs None

Additional context Related to: https://github.com/alchemyplatform/rundler/issues/496

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.