Enable replica consistency check on data movement

[sbodagala]

Enable replica consistency check on data movement and, randomly, on all reads.

Testing:

Joshua jobs:

20240524-203518-sre-1c1a2dcbba105340 (shows errors in restart tests). Another run had a single failure (on data movement not completing in "QuietDatabase", probably because replica consistency check was failing) but I don't see the problem when run locally (the test uses rocksdb, probably because of that). An update on this failure: This failure (or a similar failure) showed up on a SQLite test which we investigated. This failure was because the test was modifying a knob such that any transaction older than 1M versions would get failed by the resolver with "transaction_too_old" error. So I don't think this failure was caused by this change set.

20240613-224222-sre-b22072de0b076453: Shows a single test failure, and this test failed in a nightly run too.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• [ ] The PR has a description, explaining both the problem and the solution.
• [ ] The description mentions which forms of testing were done and the testing seems reasonable.
• [ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• [ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
• [ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

[sbodagala]

We now support the following policies on the number of replicas that the source replica will be compared against:

• BEST_EFFORT (all available storage replicas, no errors will be thrown if any of the replicas are not available)
• ALL_REPLICAS (all storage replicas, an error will be thrown if any of the replicas are not available)
• Specific number of replicas (an error will be thrown if the specified number of replicas are not available)

It appears to me this policy is asymmetric (as there is no way to specify BEST_EFFORT on a specific number of replicas). Probably we need a policy that captures the following:

• number of required replicas (this could include "ALL_STORAGE_SERVER_REPLICAS" to capture the number of replicas equal to the replication factor)
• BEST_EFFORT (no errors will be thrown if the specified number of replicas are not available) or STRICTLY_REQUIRED (an error will be thrown if the specified number of replicas are not available).

But this will require us to propagate yet another parameter to the load balancer.

Thoughts?

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 26a46818afc030414c71fe8fd722ef99be838c50
• Duration 0:26:48
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[sbodagala]

I am still a bit nervous about throwing an error on not finding the specified number of replicas (by the replica comparison actor) in fetchKeys(), because fetchKeys() would cause the storage server to fail if "fetchKeyCanRetry()" returns false on such an error. Also, blocking data movement (and making it retry) might become an issue in production as it would require a manual intervention and cause us to restart the storage servers in order to make progress on data movement in extreme scenarios (like when a single storage server is available for a given shard). Spreading the knowledge about identifying/resolving data movement not making progress to all on-call people could also become an issue. Thoughts please?

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 26a46818afc030414c71fe8fd722ef99be838c50
• Duration 0:33:24
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 26a46818afc030414c71fe8fd722ef99be838c50
• Duration 0:44:40
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 26a46818afc030414c71fe8fd722ef99be838c50
• Duration 0:46:46
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 26a46818afc030414c71fe8fd722ef99be838c50
• Duration 0:55:35
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 26a46818afc030414c71fe8fd722ef99be838c50
• Duration 0:56:08
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 1c46a5a32de864306e370a913feed9945ade5d97
• Duration 0:21:32
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 1c46a5a32de864306e370a913feed9945ade5d97
• Duration 0:34:32
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 1c46a5a32de864306e370a913feed9945ade5d97
• Duration 0:41:06
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 1c46a5a32de864306e370a913feed9945ade5d97
• Duration 0:46:04
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 1c46a5a32de864306e370a913feed9945ade5d97
• Duration 0:46:50
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 1c46a5a32de864306e370a913feed9945ade5d97
• Duration 0:50:33
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[sbodagala]

I am still a bit nervous about throwing an error on not finding the specified number of replicas (by the replica comparison actor) in fetchKeys(), because fetchKeys() would cause the storage server to fail if "fetchKeyCanRetry()" returns false on such an error. Also, blocking data movement (and making it retry) might become an issue in production as it would require a manual intervention and cause us to restart the storage servers in order to make progress on data movement in extreme scenarios (like when a single storage server is available for a given shard). Spreading the knowledge about identifying/resolving data movement not making progress to all on-call people could also become an issue. Thoughts please?

Agree blocking data movement can become an issue. So far my thought is that this is better than moving corrupted data away and accepting the fact that when only one replica is left, data movement will be blocked. So for such a scenario, we should add visibility (trace events?) for oncalls to respond/decide, e.g., deploying a knob change to disable the replica comparison if needed.

I agree with this thought at a high level - we like to block data movement if an inconsistency is detected. But if there isn't another replica available then do we still want to block (because we couldn't validate the results)? Manual intervention in this case will only result in restarting the storage servers by turning the knob off (so data movement can happen without validation), so what are we achieving by blocking the data movement in this case? I guess the question is are there any scenarios where manual intervention/otherwise would cause the failed replicas to become available (even then they might be lagging behind the latest version and so might not help with validation?) so validation could be done?

[sbodagala]

Thinking more about it, I don't think we can guarantee that data movement will get blocked (indefinitely) when an inconsistency is detected. I think it will depend on how many replicas are available and how many of them are corrupted. This is because an inconsistency would cause fetchKeys() to retry and fetchKeys() could then choose a different source. But data movement would block if a corrupted replica gets chosen as either the source or the replica to compare the source against. So it will achieve the goal of not spreading corruption (even though it is not guaranteed to block data movement, or detect data corruption, in all possible cases).

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 30a0608aa5b41b3c93ff1fcf63f521100d9ebf0f
• Duration 0:25:07
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: a56a3523daa3bc8e1a1b6e7c5f0d0ee138961509
• Duration 0:26:53
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: a56a3523daa3bc8e1a1b6e7c5f0d0ee138961509
• Duration 0:42:52
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: a56a3523daa3bc8e1a1b6e7c5f0d0ee138961509
• Duration 0:48:53
• Result: :x: FAILED
• Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 30a0608aa5b41b3c93ff1fcf63f521100d9ebf0f
• Duration 0:54:35
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: a56a3523daa3bc8e1a1b6e7c5f0d0ee138961509
• Duration 0:56:46
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 30a0608aa5b41b3c93ff1fcf63f521100d9ebf0f
• Duration 0:56:15
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 30a0608aa5b41b3c93ff1fcf63f521100d9ebf0f
• Duration 1:00:59
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: 756b99c54ced8a1a37f600f8f8a6bf998d0a5761
• Duration 0:22:06
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: 756b99c54ced8a1a37f600f8f8a6bf998d0a5761
• Duration 0:36:06
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: 756b99c54ced8a1a37f600f8f8a6bf998d0a5761
• Duration 0:47:19
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: 756b99c54ced8a1a37f600f8f8a6bf998d0a5761
• Duration 0:51:34
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: 756b99c54ced8a1a37f600f8f8a6bf998d0a5761
• Duration 1:05:42
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: 756b99c54ced8a1a37f600f8f8a6bf998d0a5761
• Duration 3:00:02
• Result: :x: FAILED
• Error: Build has timed out.
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-ide on Linux CentOS 7

• Commit ID: fa069a66444ada8c04a6960f36e0c7a1cb32b808
• Duration 0:22:34
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

• Commit ID: fa069a66444ada8c04a6960f36e0c7a1cb32b808
• Duration 0:37:02
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-macos on macOS Ventura 13.x

• Commit ID: fa069a66444ada8c04a6960f36e0c7a1cb32b808
• Duration 0:46:43
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang-arm on Linux CentOS 7

• Commit ID: fa069a66444ada8c04a6960f36e0c7a1cb32b808
• Duration 0:47:47
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

• Commit ID: fa069a66444ada8c04a6960f36e0c7a1cb32b808
• Duration 0:51:38
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)
• Cluster Test Logs zip file of the test logs (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr-clang on Linux CentOS 7

• Commit ID: fa069a66444ada8c04a6960f36e0c7a1cb32b808
• Duration 0:55:48
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[foundationdb-ci]

Result of foundationdb-pr on Linux CentOS 7

• Commit ID: fa069a66444ada8c04a6960f36e0c7a1cb32b808
• Duration 1:02:09
• Result: :white_check_mark: SUCCEEDED
• Error: N/A
• Build Log terminal output (available for 30 days)
• Build Workspace zip file of the working directory (available for 30 days)

[sbodagala]

I think this PR can be merged.