apple / foundationdb

FoundationDB - the open source, distributed, transactional key-value store
https://apple.github.io/foundationdb/
Apache License 2.0
14.19k stars 1.29k forks source link

Throw errors in getConsistentReadVersion [release-7.3] #11353

Closed jzhou77 closed 2 months ago

jzhou77 commented 2 months ago

cherrypick https://github.com/apple/foundationdb/pull/11311

In the current code, errors are retried in getConsistentReadVersion, so it's possible that the client has cancelled the GRV request, but readVersionBatcher continue retrying, which can lead to many clients DDoS GRV proxies, especially when the database has become unavailable for a while and clients are issuing many GRV requests.

Code-Reviewer Section

The general pull request guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-macos on macOS Ventura 13.x

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-cluster-tests on Linux CentOS 7

foundationdb-ci commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

foundationdb-ci commented 2 months ago

Result of foundationdb-pr-clang on Linux CentOS 7

jzhou77 commented 2 months ago

Result of foundationdb-pr on Linux CentOS 7

  • Commit ID: 105062a
  • Duration 1:20:06
  • Result: ❌ FAILED
  • Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
  • Build Log terminal output (available for 30 days)
  • Build Workspace zip file of the working directory (available for 30 days)

CI failure is not reproducible, probably because CI test timeout is shorter.