Figment logs indicate that nodes sometimes do not respond because they may be lagging behind other nodes, to the point that the entitlement was already processed by a quorum of nodes when a lagging node is starting. This change would add logging to the chain monitor to detect when an event is processed > 4s after the block occurred, as a way to determine how far behind the entitlement request a node is when it (finally) responds.
this issue should not cause the entitlement check to fail to complete, and in fact only happens as described if the check did complete
we solved the outage that was motivating this investigation. I think this only makes sense to look into at this time if our xchain stack fails to produce a check result.
Figment logs indicate that nodes sometimes do not respond because they may be lagging behind other nodes, to the point that the entitlement was already processed by a quorum of nodes when a lagging node is starting. This change would add logging to the chain monitor to detect when an event is processed > 4s after the block occurred, as a way to determine how far behind the entitlement request a node is when it (finally) responds.