On our latest deploy we moved a couple of validators to a new server, so they should've synced from scratch. It showed we still have issues with sync.
For substitutes, looks like they are both triggered and the runtime was substituted. Although on the second problematic block the syncing stucks. Behavior is different depending on the block speicified for substitution. In case it's the problematic block or earlier, the issue is "digest mismatch". In case of later blocks (like +1 block) the issue with deviding by 0 happens (which we patched by substitute). And while the first case is irreturnable failure, in the second case if we change the substitute to the problematic block (i.e. re-running the node with changed substitute after the issue happened), the node continues to sync up to the latest finalized block.
On our latest deploy we moved a couple of validators to a new server, so they should've synced from scratch. It showed we still have issues with sync.
For substitutes, looks like they are both triggered and the runtime was substituted. Although on the second problematic block the syncing stucks. Behavior is different depending on the block speicified for substitution. In case it's the problematic block or earlier, the issue is "digest mismatch". In case of later blocks (like +1 block) the issue with deviding by 0 happens (which we patched by substitute). And while the first case is irreturnable failure, in the second case if we change the substitute to the problematic block (i.e. re-running the node with changed substitute after the issue happened), the node continues to sync up to the latest finalized block.