fix: update chain when a network resets

[pdp2121]

High Level Overview of Change

When a network resets, there will be no major issue with the existing APIs (except amendments for that network has to be manually updated, which would be in a separate ticket after the amendments PR has been merged).

One minor issue with the existing process is that, if after the reset, validations come in and the chain has not been purged (purging happens every hour if the chain has not been updated for an hour, so there is still a chance that this is the case), then new ledgers would not be added into the chain for agreement calculation, hence, some agreement scores will be missed until the chain has been purged and a new chain is added. Hence, agreement score might still be calculated based on data before reset only. (Existing code for reference:

   if (chainWithThisValidator !== undefined) {
      const skipped = ledger.ledger_index - chainWithThisValidator.current
      log.warn('Possibly skipped ${skipped} ledgers')
      // all new validations after reset will not satisfy this condition and will not be added
      // until the chain is purged due to inactivity.
      if (skipped > 1 && skipped < 20) {
        chainWithThisValidator.incomplete = true
        addLedgerToChain(ledger, chainWithThisValidator)
      }
    }

    if (chainWithThisValidator !== undefined || chainWithLedger !== undefined) {
      return
    }

)

The fix for this would be:

• If the chain has not been purged, reset the chain if see a validation from overlapping validator that has ledger_index less than 20 compared to the current index the chain has seen from previous agreement calculation (one hour ago). Delete all agreement scores of all validators in the chain to recalculate.
• If the chain has been purged, when validations come in, a new chain will be created. Clears all agreement scores for all validators belong to this chain before reset.

Type of Change

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Refactor (non-breaking change that only restructures code)
• [ ] Tests (You added tests for code that already exists, or your new feature included in this PR)
• [ ] Documentation Updates
• [ ] Release

Test Plan

Tests passed. The APIs run normally on staging for a few days.

[ckniffen]

One thing I just realized is that agreement does not seem to check the hash the validator gives for a ledger and what the validated ledger hash was. Is that logic somewhere else? If it is not then we are not tracking agreement but rather that a validator is sending out validations.

[pdp2121]

We decided that the loss of data for an hour or two is a small issue, and my fix is assuming that all validators function normally (if there's a validator that is way behind on its validations or a misconfigured validator that validates on more than one networks, my fix would break and the chain would be reset, which would be an undesirable behavior). We would disregard this change and move forward as is.