[Feature Request] Design the unified download interface

[kotwanikunal]

Is your feature request related to a problem? Please describe

Segment download mechanism for Snapshot-Restore, SearchableSnapshot, RemoteStore are currently fragmented across the core code base.

Describe the solution you'd like

Design a unified handler for segment downloads across the core code base.

Related component

Search:Remote Search

Describe alternatives you've considered

No response

Additional context

No response

[kotwanikunal]

Unified Download Manager

Overview

Parallel/MultiStream downloads need to accommodate for a unified DownloadManager to perform downloads from the repository for remote indices (remote store/remote index/searchable snapshot)

Currently, these download managers are fragmented where each feature has duplicated code with separate job queue to perform the required operations.

The following sections propose a design for an unified download manager where all the jobs will be submitted to a unified queue and prioritized based on the nature of download, as well as maximize the concurrency utilization and bounds for the download thread pool(s).

Current State of Downloads

This section outlines the current flow for file download use cases in a TL;DR format.

How does the remote translog download mechanism work?

• TranslogTransferManager performs the Translog download operations for remote translog.
• downloadTranslog() and readMetadata() are the two calls utilized by remote translog to perform fetch operations from the repository.
• These calls are passed down to the repository through the BlobTransferService class which orchestrates the calls and performs the writes to disk using the BlobContainer interface.

How does the snapshot restore segment copy work?

• BlobStoreRepository handles the segment copy from the repository to the local store during shard/index restore.
• This class also handles the concurrency for the number of files to be restored(downloaded) using the appropriate thread pool and workers.
• The download mechanism utilizes a simple buffer based copy mechanism using streams to transfer data from the repository to the local store.

How does the remote store segment copy work?

• RemoteStoreFileDownloader handles the segment download process for remote store interactions.
• The behavior is similar to snapshot restore segment copy described above where RemoteStoreFileDownloader handles the concurrency and uses a similar buffer based mechanism to copy data from the remote store to a local store.

How does the searchable snapshot block fetching work?

• The read-only engine hits the store for segment access, trying to first locate it in the file cache.
• In case of a cache miss - the class OnDemandBlockSnapshotIndexInput fetches it from the snapshot stored in the repository.
• It uses the TransferManager to download the required block for the segment on disk, adds an entry in the cache and returns the block to the engine for query completion.

Proposal

A new, unified DownloadManager will help with the remote index (remote store/remote index/searchable snapshot) based requests.

• All the use cases currently utilize their own implementations of an orchestrator/mediator (TransferManager for Searchable Snapshots, RemoteStoreFileDownloader for Remote Store)
• The new Download``Manager class will be responsible for orchestrating, queuing, tracking, retrying file downloads from the Repository APIs defined above for all the fetch requests and will also take care of the concurrency aspects for the download manager including ensuring the use of appropriate ThreadPool sizing configurations, file parallelism v/s stream parallelism (possibly using PriorityQueue(s))
• The DownloadManager will also be additionally responsible for RemoteTranslog operations which are performed using the Repository interface.

Design

• The unified DownloadManager will be developed in phases as described in the following section.
• The implementation can use an existing implementation of an executor which will be initialized as a part of the ThreadPool initialization
  • PrioritizedOpenSearchThreadPoolExecutor accepts a Runnable as a part of the execute call
  • The wrappedRunnable will have an associated Priority with it which will be supplied at the time of job creation within the new DownloadManager
• The designed unified queue will take into account the RecoverySettings for concurrency between the various download operations, and limit it using a separate PriorityQueue instantiation in DownloadManager

APIs

APIs for DownloadManager will evolve over a few phases -

Phase 1: Introduction of a new, unified DownloadManager

• Phase 1 will involve wiring up the DownloadManager across the different use cases and unifying the code into a single class.

Phase 2: Introduction of standard, re-usable APIs

• Phase 2 will be an evolution and refinement of APIs where they would be generalized to perform the operations like downloadFile, downloadBlock
• This phase will require a redesign to move the operation specific code back to the calling classes and/or a separate wrapper class.

Phase 3: Introduction of a unified executor queue for download orchestration

• The download calls/APIs will be converted to use an executable interface - representing individual jobs and these jobs would be submitted to the unified queue (PrioritizedOpenSearchThreadPoolExecutor ), also honoring the RecoverySetting configuration/queue management,