Upgrade AWS SDK v1 to v2

[dblock]

Description

Upgrading AWS SDK v1 to v2 in discovery-ec2 and repository-s3.

This PR contains the following potentially breaking changes for end-users.

• The AWS SDK v2 uses aws.ec2MetadataServiceEndpoint instead of com.amazonaws.sdk.ec2MetadataServiceEndpointOverride. If a user set the IMDS endpoint before that will no longer work without a change. Similarly we've renamed com.amazonaws.sdk.stsEndpointOverride to aws.stsEndpoint. What are the use-cases where anyone would actually do this, or was it done for testing only?
• AWS SDK v2 introduced a concept of TokenBucketRetryCondition, which effectively limits retries on a bucket level for APIs that have been failing continuously. It allocates tokens (default is 500), which means that once 500 retries fail for any API on a bucket, new retries will only be allowed once some retries are rejected.

The AWS SDK has other changes, see https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/migration-client-changes.html and https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md for all changes between AWS SDK v1 and v2.

Related Issues

Resolves #7312.

Check List

• [x] New functionality includes testing.
  • [x] All tests pass
• [x] New functionality has been documented.
  • [x] New functionality has javadoc added
• [x] Commits are signed per the DCO using --signoff
• [x] Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/14756/
• CommitID: b2e2d472f326c3a8ff5559c197b18d890eebd855 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/14757/
• CommitID: fdf8453aecbccfc9da459f1082d7c57728d73ee2 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[Bukhtawar]

Code scan for dependencies, looks like we have just the two plugins consuming AWS SDK

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15036/
• CommitID: 2b3980f4c41c0206a9f11fd8f32f050509d50b74 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15116/
• CommitID: 29f23697de48f03bdc89ff9bb2dec95fdbbf0b88 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15117/
• CommitID: b182eedff3dece3272433df356511cc8ef0d393d Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15397/
• CommitID: 492215a22be36ccd7e9b00e399da91f1febc67ce Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15418/
• CommitID: d79e658322fde23d531c63db716905b3d71cb88f Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15427/
• CommitID: f6b3a4ce96134606b4354170eb3db9e14e518b12 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15432/
• CommitID: 0d501b310579bd8632273547f0da84bb0a8df5de Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15445/
• CommitID: 24d1eb85f775ad79ce006f8dfb2434abb7ce34ae Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15460/
• CommitID: aec884d86010e1df4c9a5c40c263888f985c28fa Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15472/
• CommitID: 9461ac2d0e02b5d3b362de045f665674e0d9a128 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15475/
• CommitID: b4731c8211464e3ffc75a57e2cb55e3af6bfa83d Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15500/
• CommitID: 053b0c0fd1197a539f1420687ed61e9a760e8566 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15564/
• CommitID: c754b0d393db01ec14f3a9d63ff8ffcbbeb4e4c0 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15565/
• CommitID: 1ab9dc7bb39ee9583cce56e948206b31c9890779 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15717/
• CommitID: b45b05abe82122f2b69b298261c0d2b5f31892ae Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15722/
• CommitID: cc60688a895618d7cddf1a7fabe44f200bcf4554 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15738/
• CommitID: 3bcaff0acf455fa7c6894690c4e7a0ec273270f6 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15763/
• CommitID: a9b68c7b98ce397ae3c1f9246d3ea46c42a76c9a Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15765/
• CommitID: ad15dbe9880a776df5aa1c5488005986d0879c3a Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15825/
• CommitID: faa33254e51440313d5e1f2341ae2fb55d2c99c6 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15827/
• CommitID: faa33254e51440313d5e1f2341ae2fb55d2c99c6 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15843/
• CommitID: 486a01c28317a97f4d2393ce72569187a232269e Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

Looks like :plugins:discovery-ec2:qa:amazon-ec2:yamlRestTestSystemProperties is failing.

»  software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Secret key must be specified either via environment variable (AWS_SECRET_ACCESS_KEY) or system property (aws.secretAccessKey)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(profilesAndSectionsMap=[])): Profile file contained no credentials for profile 'default': ProfileFile(profilesAndSectionsMap=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
»   at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[sdk-core-2.20.55.jar:?]
»   at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:117) ~[auth-2.20.55.jar:?]
»   at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) ~[auth-2.20.55.jar:?]
»   at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:126) ~[auth-2.20.55.jar:?]
»   at software.amazon.awssdk.core.internal.util.MetricUtils.measureDuration(MetricUtils.java:50) ~[sdk-core-2.20.55.jar:?]
»   at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.resolveCredentials(AwsCredentialsAuthorizationStrategy.java:100) ~[aws-core-2.20.55.jar:?]
»   at software.amazon.awssdk.awscore.internal.authcontext.AwsCredentialsAuthorizationStrategy.addCredentialsToExecutionAttributes(AwsCredentialsAuthorizationStrategy.java:77) ~[aws-core-2.20.55.jar:?]
»   at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:123) ~[aws-core-2.20.55.jar:?]
»   at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:69) ~[aws-core-2.20.55.jar:?]
»   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:78) ~[sdk-core-2.20.55.jar:?]
»   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:179) ~[sdk-core-2.20.55.jar:?]
»   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:76) ~[sdk-core-2.20.55.jar:?]
»   at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.20.55.jar:?]
»   at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:56) ~[aws-core-2.20.55.jar:?]
»   at software.amazon.awssdk.services.ec2.DefaultEc2Client.describeInstances(DefaultEc2Client.java:18739) ~[ec2-2.20.55.jar:?]
»   at org.opensearch.discovery.ec2.AwsEc2SeedHostsProvider.lambda$fetchDynamicNodes$0(AwsEc2SeedHostsProvider.java:134) ~[discovery-ec2-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at java.security.AccessController.doPrivileged(AccessController.java:318) ~[?:?]
»   at org.opensearch.discovery.ec2.SocketAccess.doPrivileged(SocketAccess.java:63) ~[discovery-ec2-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.discovery.ec2.AwsEc2SeedHostsProvider.fetchDynamicNodes(AwsEc2SeedHostsProvider.java:134) [discovery-ec2-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.discovery.ec2.AwsEc2SeedHostsProvider$TransportAddressesCache.refresh(AwsEc2SeedHostsProvider.java:258) [discovery-ec2-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.discovery.ec2.AwsEc2SeedHostsProvider$TransportAddressesCache.refresh(AwsEc2SeedHostsProvider.java:250) [discovery-ec2-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.common.util.SingleObjectCache.getOrRefresh(SingleObjectCache.java:68) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.discovery.ec2.AwsEc2SeedHostsProvider.getSeedAddresses(AwsEc2SeedHostsProvider.java:118) [discovery-ec2-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.discovery.DiscoveryModule.lambda$new$4(DiscoveryModule.java:182) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.discovery.SeedHostsResolver$1.doRun(SeedHostsResolver.java:275) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:806) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) [opensearch-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
»   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
»   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
»   at java.lang.Thread.run(Thread.java:1589) [?:?]
»   ↑ repeated 7 times ↑

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/15846/
• CommitID: 92e618a174754fb1fb79e187dbaeb7271d6aa7d2 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS :white_check_mark:
• URL: https://build.ci.opensearch.org/job/gradle-check/15847/
• CommitID: 9179110d3f0901ba3d3a57d7da0d0fcd8ac7adf4

[codecov[bot]]

Codecov Report

Merging #7372 (a08cb1f) into main (ecd0879) will increase coverage by 0.16%. The diff coverage is 76.49%.

@@             Coverage Diff              @@
##               main    #7372      +/-   ##
============================================
+ Coverage     70.74%   70.91%   +0.16%     
- Complexity    56201    56333     +132     
============================================
  Files          4687     4689       +2     
  Lines        266242   266340      +98     
  Branches      39096    39098       +2     
============================================
+ Hits         188364   188870     +506     
+ Misses        61905    61505     -400     
+ Partials      15973    15965       -8     

Impacted Files	Coverage Δ
...va/org/opensearch/discovery/ec2/AwsEc2Service.java	78.57% <ø> (ø)
...g/opensearch/discovery/ec2/Ec2DiscoveryPlugin.java	76.08% <0.00%> (-3.92%)	:arrow_down:
...opensearch/repositories/s3/S3RepositoryPlugin.java	0.00% <ø> (-26.93%)	:arrow_down:
.../org/opensearch/repositories/s3/ProxySettings.java	70.00% <50.00%> (+7.50%)	:arrow_up:
...search/common/concurrent/RefCountedReleasable.java	90.00% <50.00%> (-10.00%)	:arrow_down:
...ensearch/repositories/s3/utils/HttpRangeUtils.java	57.14% <57.14%> (ø)
...ensearch/repositories/s3/StatsMetricPublisher.java	59.25% <59.25%> (ø)
...nsearch/discovery/ec2/AwsEc2SeedHostsProvider.java	65.97% <66.66%> (+2.53%)	:arrow_up:
...java/org/opensearch/repositories/s3/S3Service.java	75.26% <74.50%> (-5.11%)	:arrow_down:
.../opensearch/repositories/s3/AmazonS3Reference.java	91.66% <75.00%> (ø)
... and 12 more

... and 429 files with indirect coverage changes

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16035/
• CommitID: 8d86d4d126cd8d072288ec7bd7a995f6e50bae10 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[Bukhtawar]

Looks like I approved only the EC2 changes there are more commits following in. Can we review both EC2 and S3 changes independently and merge finally?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16039/
• CommitID: b81de739f6839e92243597e5c606be61b9916ffd Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[dblock]

Looks like I approved only the EC2 changes there are more commits following in. Can we review both EC2 and S3 changes independently and merge finally?

@raghuvanshraj and I decided to combine the two in 1 PR because the changes are quite similar in nature and we want to make sure we don't diverge across one or the other plugin (e.g. for override settings or how proxy is setup). There's some breaking changes that will need to be discussed together as well that affect both plugins.

The changes between the 2 plugins are isolated though, so you can look in subfolders in 1 PR.

Will mark this PR ready as soon as we've resolved all outstanding issues.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16047/
• CommitID: 662895f1e400655b812469e8794c251cbeb9f4ba Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16061/
• CommitID: 55c71a29b9b8bdd4dc04630940529421f0a0ac79 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE :grey_exclamation:
• TEST FAILURES:

    1 org.opensearch.cluster.allocation.AwarenessAllocationIT.testThreeZoneOneReplicaWithForceZoneValueAndLoadAwareness

• URL: https://build.ci.opensearch.org/job/gradle-check/16072/
• CommitID: 01ff04e2e146b10513f7c6f17c309cf1f9bea46f Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE :grey_exclamation:
• TEST FAILURES:

    1 org.opensearch.indices.replication.SegmentReplicationRelocationIT.testPrimaryRelocation

• URL: https://build.ci.opensearch.org/job/gradle-check/16075/
• CommitID: 765ac7c21d88e1440768ccd56bcac4451706c131 Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE :grey_exclamation:
• TEST FAILURES:

    1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=tasks.list/10_basic/tasks_list test}
    1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

• URL: https://build.ci.opensearch.org/job/gradle-check/16136/
• CommitID: d177a8b56d1d0ea67e4be290aa8b62846ea9a4b4 Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16174/
• CommitID: 1002207b4ef3a3f099a0e88fb2ecc5c35517c4ac Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE :grey_exclamation:
• TEST FAILURES:

    1 org.opensearch.backwards.MixedClusterClientYamlTestSuiteIT.test {p0=tasks.list/10_basic/tasks_list test}

• URL: https://build.ci.opensearch.org/job/gradle-check/16182/
• CommitID: 7531f1f86fd2b0c919a48838d9cec7ef4566d08a Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[dblock]

@Bukhtawar care to do a pass on this PR?

[andrross]

I'm not super familiar with the integration testing here. I see we have some test fixtures, but in my experience test fixtures rarely offer perfect fidelity with the real services. Is there any testing beyond the tests that already run that we should do?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16320/
• CommitID: 13d8d77ac7284009b8ee4c617678b9f166bda4c0 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16321/
• CommitID: 7531f1f86fd2b0c919a48838d9cec7ef4566d08a Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16324/
• CommitID: a70c99ea0dc536b4f70b010477234e7febf956cc Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16323/
• CommitID: 651a61b26968062c647c86597e9516868a4e1658 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[reta]

Massive one, @dblock :clap: :) did first pass

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE :grey_exclamation:
• TEST FAILURES:

    1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchTaskCancellationWithHighCpu

• URL: https://build.ci.opensearch.org/job/gradle-check/16322/
• CommitID: 2634172196a272eccdfe36e24c5d55a476140c57 Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16352/
• CommitID: 66ddc414c3fd12398d59fd5938a4be3c4dd9d4b7 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE :x:
• URL: https://build.ci.opensearch.org/job/gradle-check/16353/
• CommitID: c02fb11c043608f7775544563691db18ad8ca883 Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?