[Remote Store] Add segment transfer timeout dynamic setting

linuxpi commented 2 weeks ago

Description

Segment uploads for Remote Store happen without any timeout and can hold the thread on latch.await() for ever if there is an error which is swallowed silently by the code.
Since the future is never completed, the shard is stuck and is never able to retry the upload
This leads to remote time lag to increase and never come down until we restart the process
This PR adds the timeout to segment uploads to recover automatically from such situations after the timeout.

Check List

[x] New functionality includes testing.
- [x] All tests pass
[x] New functionality has been documented.
- [x] New functionality has javadoc added
[ ] ~API changes companion pull request created.~
[ ] ~Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)~
[x] Commits are signed per the DCO using --signoff
[x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)
[x] Public documentation issue/PR created https://github.com/opensearch-project/documentation-website/issues/7227

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions[bot] commented 2 weeks ago

:x: Gradle check result for 8f59a8d58f79fe5a0f8f93ebd210b3dca169abb1: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 2 weeks ago

:x: Gradle check result for 85416eb908b69769a1a0c7bcdae2a79e4df085cc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for a4ac198831e72b9351a47415497c32e7379bbdd0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for 16eb7d2e3e3b6813944e15ebe7232fda01d5cc69: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for 0fd7f83282f71673b1edd517849f95c83fffbb2f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for 4b67fa0777a174480e5a2d15a568396539670d39: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for f4546d7fd6390ed6ea3504d3ea7cc3e9bf45663d: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for 2794e7b5cc694a6fe08a8c12549608706a66e6e4: null

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for 783b0ef7a5b4f91b2319dd0fa8f49ff15168967b: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for 62e73e52120ed3677a73bf310b2aba24d59e0af0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:x: Gradle check result for cbca4eea6e1a3ae83339c8af865d6a9e6e1111af: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions[bot] commented 1 week ago

:white_check_mark: Gradle check result for a6417b473e68dcb18d8bd8010fe89d0057ff9199: SUCCESS

codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 86.66667% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 71.58%. Comparing base (b15cb0c) to head (251ecd0). Report is 299 commits behind head on main.

Files	Patch %	Lines
...va/org/opensearch/indices/RemoteStoreSettings.java	75.00%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #13679 +/- ## ============================================ + Coverage 71.42% 71.58% +0.16% - Complexity 59978 61255 +1277 ============================================ Files 4985 5063 +78 Lines 282275 287917 +5642 Branches 40946 41691 +745 ============================================ + Hits 201603 206108 +4505 - Misses 63999 64770 +771 - Partials 16673 17039 +366 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

sachinpkale commented 1 week ago

Please add changelog entry, fix the gradle build and complete the check-list in PR description.

linuxpi commented 1 week ago

Please add changelog entry, fix the gradle build and complete the check-list in PR description.

Thanks @sachinpkale for approving. I've added the necessary details and addressed your comment.

github-actions[bot] commented 1 week ago

:grey_exclamation: Gradle check result for 251ecd08d9f959bd2327a0272d1640644c507468: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

opensearch-trigger-bot[bot] commented 1 week ago

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-13679-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 b3049fb5ec860f63fdfe33c5b176869b1e4255e6
# Push it to GitHub
git push --set-upstream origin backport/backport-13679-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-13679-to-2.x.

opensearch-project / OpenSearch