[Meta] Remote store-based warm index

[andrross]

Goals

Create a proof-of-concept that shows end-to-end functionality of a remote-backed index where the data may not all reside locally and can be fetched on-demand from the remote store when necessary. This is the initial implementation of the feature described in #6528. This will build upon the design and prototype started in #7331 in order to demonstrate an end-to-end capability.

This code touches much of the same code as the remote store feature, which is nearing promotion out from behind a feature flag. In order to avoid complicating that effort in the immediate short term, we’ll start development on a feature branch. Once remote store is no longer behind a feature flag, then we’ll move this effort from the feature branch to behind a feature flag on main.

Non-goals

Make final decisions on naming or APIs. The term “warm” is used extensively here as that is a sort of term-of-art and is generally well understood, but one of the larger goals of these efforts is to remove the need for users to think about discrete storage tiers and allow the system to more intelligently optimize based on usage patterns.

Tasks

• [x] #8447
• [ ] #8448
• [ ] #8449
• [ ] #8450
• [x] #8802
• [ ] #8803
• [ ] #8804

---

The above tasks are the initial priority for building the basic functionality. After that, we will implement the functionality described below to dynamically change the "warm" property on an index:

• Implement support for adding “warm” setting to hot index (hot-to-warm)
  • All files on local disk are logically moved to the file cache (no physical movement). If the file cache is filled, then files will be deleted from local disk based on a TBD heuristic.
  • The index will remain readable and writable with no extended downtime.
• Implement support for removing “warm” setting from warm index (warm-to-hot)
  • All block-based files will be deleted from disk and complete files will be restored from the remote store.
  • The index will remain readable and writable with no extended downtime.

[anasalkouz]

All block-based files will be deleted from disk and complete files will be restored from the remote store.

Why this is required since we will have the hybrid directory and we can read from both complete files and block-based files?

[ankitkala]

Issues for FileCache+Block level fetch(apart from the ones already mentioned in the description):

• [ ] https://github.com/opensearch-project/OpenSearch/issues/11789
• [ ] https://github.com/opensearch-project/OpenSearch/issues/11790
• [ ] https://github.com/opensearch-project/OpenSearch/issues/11791
• [ ] https://github.com/opensearch-project/OpenSearch/issues/11792
• [ ] https://github.com/opensearch-project/OpenSearch/issues/11793

[ankitkala]

Here is the sorted list of tasks we have to start the efforts on writable warm.

• Add initial bootstrap code for warm primary (issue1 issue2).
  • Uses file cache & composite directory.
  • Implement IndexInput for local files with file cache.
• Support for replicas (issue).
  • Replica recovery flow (start replica with partial data on remote)
  • Supporting segment replication on warm replicas.
  • Prefetch optimizations for replica (data needed to open a reader).
  • Add support for remote store backed index inputs.
• Primary promotion (replica to primary, primary relocation) (issue).
  • Testing write path on primary with tiered data.
• Support encryption with writable warm(issue)
• Prefetch optimizations (TBD).
• Support for File cache evictions. TTL & overflow criteria based.
  • open questions: how long to retain the new files locally on primary.
  • Overflow criteria for File Cache.
• [Observability] Expose stats & metrics for FileCache (issue).