Top-down events consistency checks

[raulk]

Context

We currently blindly trust the Ethereum events and receipts returned by the RPC endpoint when importing parent finality. If the RPC endpoint returns no events, we accept that blindly. However, Lotus makes no guarantees of preserving receipts, and events are returned from a cache that has no consistency guarantees with the chain itself. So relying solely on this is extremely brittle.

Solution

Implement consistency checks. When importing finality, perform a diff between the current and previous configuration number and top-down message nonces at the parent subnet actor and gateway, and reject accepting the finality if the RPC endpoint does not return the expected events.

This will prevent us from importing incomplete parent finality by blocking the whole process and prioritising consistency over progress. We should print a warning/alert, emit an event, and increment a Prometheus metric when this happens.

_{From SyncLinear.com | ENG-835}

[raulk]

From @aakoshh:

I thought the nonce checks are already done when adding messages to the cache.

I'm not sure if you can reliably use consistency checks if all nodes are pointed at RPC endpoints which simply don't serve data, but they do this consistently. If the data isn't available to anybody, then the system is simply not fit for purpose. The operators should probably run their own nodes on the parent which make the data available to them, rather than rely on external sources with arbitrary configurations.