[Bugfix] Fix TieredSpilloverCache stats not adding correctly when shards are closed

[peteralfonsi]

Description

Fixes a bug where the total stats for the TieredSpilloverCache are decremented incorrectly when shards were closed. Misses and evictions from both the heap and disk tier were subtracted from the total, but this is incorrect. When the disk tier is enabled, only disk-tier misses and evictions should count towards the cache total, so only they should be subtracted. Adds UTs and ITs around this.

Also adds UT coverage for TieredSpilloverCacheStatsHolder, which was missing before.

Related Issues

Resolves https://github.com/opensearch-project/OpenSearch/issues/16559

Check List

• [x] Functionality includes testing.
• [N/A] API changes companion pull request created, if applicable.
• [N/A] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions[bot]]

:x: Gradle check result for c2a05d4ebfcd7471523679047fedb9dc1ed28e23: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for f85b719ec58b7644d2b4d94581d7144498472cc0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for a39d586683eff077631fc216fc91b0287e14b961: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:x: Gradle check result for 165850e06d30b243713bf0f93011370572d55a15: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:grey_exclamation: Gradle check result for dbd094736d65300244cb5b6f5333697baa85aa96: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 83.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.18%. Comparing base (b1a7743) to head (2e8ffee).

Files with missing lines	Patch %	Lines
...search/cache/common/tier/TieredSpilloverCache.java	75.00%	0 Missing and 1 partial :warning:
...e/common/tier/TieredSpilloverCacheStatsHolder.java	83.33%	0 Missing and 1 partial :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #16560 +/- ## ============================================ + Coverage 72.16% 72.18% +0.02% + Complexity 65246 65225 -21 ============================================ Files 5318 5318 Lines 303940 303945 +5 Branches 43976 43976 ============================================ + Hits 219326 219412 +86 + Misses 66625 66520 -105 - Partials 17989 18013 +24 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests

[github-actions[bot]]

:x: Gradle check result for 0de62845358ecf8efb49adc9d62ef40d0a783279:

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions[bot]]

:white_check_mark: Gradle check result for 2e8ffee7992c886f814bc9903151a39b232086b6: SUCCESS