Open kotwanikunal opened 8 months ago
Parallel/MultiStream downloads need to accommodate for a unified DownloadManager to perform downloads from the repository for remote indices (remote store/remote index/searchable snapshot)
Currently, these download managers are fragmented where each feature has duplicated code with separate job queue to perform the required operations.
The following sections propose a design for an unified download manager where all the jobs will be submitted to a unified queue and prioritized based on the nature of download, as well as maximize the concurrency utilization and bounds for the download thread pool(s).
This section outlines the current flow for file download use cases in a TL;DR format.
TranslogTransferManager
performs the Translog download operations for remote translog.downloadTranslog()
and readMetadata()
are the two calls utilized by remote translog to perform fetch operations from the repository.BlobTransferService
class which orchestrates the calls and performs the writes to disk using the BlobContainer
interface.BlobStoreRepository
handles the segment copy from the repository to the local store during shard/index restore.RemoteStoreFileDownloader
handles the segment download process for remote store interactions.RemoteStoreFileDownloader
handles the concurrency and uses a similar buffer based mechanism to copy data from the remote store to a local store.OnDemandBlockSnapshotIndexInput
fetches it from the snapshot stored in the repository.TransferManager
to download the required block for the segment on disk, adds an entry in the cache and returns the block to the engine for query completion.A new, unified DownloadManager
will help with the remote index (remote store/remote index/searchable snapshot) based requests.
TransferManager
for Searchable Snapshots, RemoteStoreFileDownloader
for Remote Store)Download``Manager
class will be responsible for orchestrating, queuing, tracking, retrying file downloads from the Repository
APIs defined above for all the fetch requests and will also take care of the concurrency aspects for the download manager including ensuring the use of appropriate ThreadPool
sizing configurations, file parallelism v/s stream parallelism (possibly using PriorityQueue
(s)
)DownloadManager
will also be additionally responsible for RemoteTranslog
operations which are performed using the Repository
interface. DownloadManager
will be developed in phases as described in the following section.PrioritizedOpenSearchThreadPoolExecutor
accepts a Runnable
as a part of the execute
callwrappedRunnable
will have an associated Priority
with it which will be supplied at the time of job creation within the new DownloadManager
RecoverySettings
for concurrency between the various download operations, and limit it using a separate PriorityQueue
instantiation in DownloadManager
APIs for DownloadManager
will evolve over a few phases -
DownloadManager
across the different use cases and unifying the code into a single class.downloadFile,
downloadBlock
Phase 3: Introduction of a unified executor queue for download orchestration
PrioritizedOpenSearchThreadPoolExecutor
), also honoring the RecoverySetting
configuration/queue management,
Is your feature request related to a problem? Please describe
Segment download mechanism for
Snapshot-Restore
,SearchableSnapshot
,RemoteStore
are currently fragmented across the core code base.Describe the solution you'd like
Design a unified handler for segment downloads across the core code base.
Related component
Search:Remote Search
Describe alternatives you've considered
No response
Additional context
No response