apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store

https://apple.github.io/foundationdb/

Apache License 2.0

14.19k stars 1.29k forks source link

Improve distributed consistency checker #11346

Closed kakaiu closed 2 months ago

kakaiu commented 2 months ago

This PR includes two improvements to DCC:

ConsistencyCheckerUrgent repeatedly runs by default
ConsistencyCheckerUrgent traces SevError for TesterRecruitmentTimeout only if the situation keeps for at least 1 day.

100K correctness test with 2 irrelevant failures: 20240430-044337-zhewang-37c6028d4bed2dcc compressed=True data_size=36837578 duration=7886375 ended=100000 fail=2 fail_fast=10 max_runs=100000 pass=99998 priority=100 remaining=0 runtime=1:08:11 sanity=False started=100000 stopped=20240430-055148 submitted=20240430-044337 timeout=5400 username=zhewang

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

[ ] The PR has a description, explaining both the problem and the solution.
[ ] The description mentions which forms of testing were done and the testing seems reasonable.
[ ] Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

[ ] This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or main if this is the youngest branch)
[ ] There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 053546cec1d187a89240f974167661a2bd77b8b5
Duration 0:21:40
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 463998e363cf696463805cca43f15b5120a3c7f2
Duration 0:08:43
Result: :x: FAILED
Error: reference not found for primary source and source version 463998e363cf696463805cca43f15b5120a3c7f2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 463998e363cf696463805cca43f15b5120a3c7f2
Duration 0:08:51
Result: :x: FAILED
Error: reference not found for primary source and source version 463998e363cf696463805cca43f15b5120a3c7f2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 463998e363cf696463805cca43f15b5120a3c7f2
Duration 0:08:59
Result: :x: FAILED
Error: reference not found for primary source and source version 463998e363cf696463805cca43f15b5120a3c7f2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 463998e363cf696463805cca43f15b5120a3c7f2
Duration 0:09:01
Result: :x: FAILED
Error: reference not found for primary source and source version 463998e363cf696463805cca43f15b5120a3c7f2
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 053546cec1d187a89240f974167661a2bd77b8b5
Duration 0:34:44
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: bb8ba156e4a2db51adef5c6eca26e514baeea3b3
Duration 0:23:52
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 053546cec1d187a89240f974167661a2bd77b8b5
Duration 0:46:19
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 053546cec1d187a89240f974167661a2bd77b8b5
Duration 0:55:09
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 053546cec1d187a89240f974167661a2bd77b8b5
Duration 1:17:44
Result: :x: FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: bb8ba156e4a2db51adef5c6eca26e514baeea3b3
Duration 0:56:00
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: bb8ba156e4a2db51adef5c6eca26e514baeea3b3
Duration 0:56:24
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 053546cec1d187a89240f974167661a2bd77b8b5
Duration 1:24:54
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: bb8ba156e4a2db51adef5c6eca26e514baeea3b3
Duration 1:02:47
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Duration 0:08:29
Result: :x: FAILED
Error: reference not found for primary source and source version 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Duration 0:08:44
Result: :x: FAILED
Error: reference not found for primary source and source version 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Duration 0:08:46
Result: :x: FAILED
Error: reference not found for primary source and source version 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Duration 0:08:53
Result: :x: FAILED
Error: reference not found for primary source and source version 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Duration 0:10:35
Result: :x: FAILED
Error: reference not found for primary source and source version 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Duration 0:10:47
Result: :x: FAILED
Error: reference not found for primary source and source version 0f7d72d67763abb739f18fcbfb13c2a1fbcb43a8
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: d8c8d6ff8d17dc596766024ce1293dd705c40498
Duration 0:21:10
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: d8c8d6ff8d17dc596766024ce1293dd705c40498
Duration 0:54:42
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: d8c8d6ff8d17dc596766024ce1293dd705c40498
Duration 1:06:30
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: d8c8d6ff8d17dc596766024ce1293dd705c40498
Duration 1:10:26
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 638680d3a148244b857f59c3a9d5d61c5b6f7b4d
Duration 0:20:57
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 638680d3a148244b857f59c3a9d5d61c5b6f7b4d
Duration 0:35:21
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 638680d3a148244b857f59c3a9d5d61c5b6f7b4d
Duration 0:45:57
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 638680d3a148244b857f59c3a9d5d61c5b6f7b4d
Duration 0:54:18
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 638680d3a148244b857f59c3a9d5d61c5b6f7b4d
Duration 1:06:37
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 638680d3a148244b857f59c3a9d5d61c5b6f7b4d
Duration 1:09:54
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:21:37
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:36:09
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:47:30
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:55:21
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 1:16:05
Result: :x: FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 1:22:47
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

kakaiu commented 2 months ago

Since the failed CI test is not relevant, we re-kick the CI test.

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang-ide on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:21:26
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:34:10
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:46:51
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 0:54:33
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 1:04:36
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

Commit ID: 984bcbb2fd35b5f7889c1fdb571d3c831e0955e0
Duration 1:07:40
Result: :white_check_mark: SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)