This is a common problem we had to face in the beamer project as well. RPC providers hide there nodes behind a load balancer. This can lead to the fact two requests are made on a different state, because nodes are not in sync yet.
Specifically this problem arises at the following algorithm. We listen to new heads via web sockets and once a new block header arrived, we immediately fetch logs for this block. The problem is that if we do these requests too quickly, then the block header might be fetched by one node while for the next request, the block hasn't seen by the assigned node.
Even worse, some RPC provider do not follow the Ethereum spec and do not return an error if fetching logs for an unknown block, instead they return an empty array, which is ambiguous to if there were no logs in this block.
I don't recall how we solved the problem in the end, but we have to figure out to not miss events for a given block. As long as the RPC provider follows the spec we can rely on the error message and retry upon error.
We also should think about if this is a problem at other places in the code base.
This is a common problem we had to face in the beamer project as well. RPC providers hide there nodes behind a load balancer. This can lead to the fact two requests are made on a different state, because nodes are not in sync yet.
Specifically this problem arises at the following algorithm. We listen to new heads via web sockets and once a new block header arrived, we immediately fetch logs for this block. The problem is that if we do these requests too quickly, then the block header might be fetched by one node while for the next request, the block hasn't seen by the assigned node.
Even worse, some RPC provider do not follow the Ethereum spec and do not return an error if fetching logs for an unknown block, instead they return an empty array, which is ambiguous to if there were no logs in this block.
I don't recall how we solved the problem in the end, but we have to figure out to not miss events for a given block. As long as the RPC provider follows the spec we can rely on the error message and retry upon error.
We also should think about if this is a problem at other places in the code base.