Paranoid durability - basic functionality

[litingulfs]

This PR adds the basic required functionality for paranoid durability, where we require PUTs to succeed in at least one remote colo, reducing the risk of data loss in exceptional situations.

The writes in the local colo follow the same logic as before, but the remote (i.e. cross colo) writes are executed as follows:

• We allow simultaneous local and remote writes (respecting the parallelism configurations).
• We prefer to write to remote replicas that are in Helix LEADER state.
• We alternate writing to each remote data center. So, for example, if the current (local) colo is prod-ltx1, then we might attempt to write first to the LEADER in prod-lor1. If that write fails, we'll try to write to the LEADER in prod-lva1 instead of attempting further replicas in prod-lor1. The assumption behind this is that if a remote colo write fails, it is likely that we have hit a networking or colo-related issue, and so we should try the other colo now.

I've tested this locally, and as agreed with the rest of the team I'll follow up with pull requests with tests and metrics and the required wiring.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 2.05479% with 143 lines in your changes missing coverage. Please review.

Project coverage is 70.08%. Comparing base (52ba813) to head (ab255f3). Report is 20 commits behind head on master.

Files	Patch %	Lines
...bry/router/ParanoidDurabilityOperationTracker.java	0.00%	143 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## master #2794 +/- ## ============================================ + Coverage 64.24% 70.08% +5.83% - Complexity 10398 11683 +1285 ============================================ Files 840 841 +1 Lines 71755 72126 +371 Branches 8611 8682 +71 ============================================ + Hits 46099 50547 +4448 + Misses 23004 18931 -4073 + Partials 2652 2648 -4 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[litingulfs]

The more I think about this, the more i feel that we shouldn't differentiate remote dcs, We should just put all remote replicas in a list. After local dc succeeds, we should just iterate through the remote list. It would probably be a bit cleaner this way.

Yeah, we spoke about this offline, and I've changed the code so that instead of tracking replicas on a DC-by-DC basis, we just track local vs remote replicas. There is a slight cost to this, since we now end up doing more work to sort the remote replicas so that we can alternate between colos and also prefer LEADER replicas, but I think overall the code is now easier to understand.