Is your feature request related to a problem? Please describe
This is an optimization related to the Writeable Warm feature.
When the writeable warm feature is introduced we will have the ability to create indices and then migrate them to a warm tier. A use case this does not cover is to create a warm index from a snapshot without having to go through expensive segment download/re-upload as a remote backed index.
Describe the solution you'd like
I think we can make this happen with a new RemoteDirectory implementation that conditionally fetches from a blob store wired to an existing snapshot or another with the remote store directory. This new dir could be injected into RemoteSegmentStoreDirectory as its data directory. All metadata continues to push to the remote store as normal it is only when fetching a file that we would interface with the original snapshot if necessary. In a way its similar to a searchable snapshot dir however those code paths would not be reusable with the incoming writeable warm CompositeDirectory implementation. That dir handles block fetch above RemoteSegmentStoreDirectory.
This would look something like below with new writes would push to the remote store as normal and reads flowing through FilteredRemoteDirectory.
A requirement here would be to enforce some level of deletion protection on the original snapshot for the lifetime of the index or at least until all segments from the original snapshot are merged away. We could do this with some new index level settings to validate at snapshot deletion time to ensure its not backing any existing index, similar to searchable snapshots.
Related component
Storage:Remote
Describe alternatives you've considered
Nothing - restore from snapshot as a hot index then migrate.
Migrate data off cluster and somehow wire it up when the dir initializes. This is risky because remote store paths are determined at index creation.
peternied
commented
2 weeks ago
Is your feature request related to a problem? Please describe
This is an optimization related to the Writeable Warm feature.
When the writeable warm feature is introduced we will have the ability to create indices and then migrate them to a warm tier. A use case this does not cover is to create a warm index from a snapshot without having to go through expensive segment download/re-upload as a remote backed index.
Describe the solution you'd like
I think we can make this happen with a new
RemoteDirectoryimplementation that conditionally fetches from a blob store wired to an existing snapshot or another with the remote store directory. This new dir could be injected into RemoteSegmentStoreDirectory as its data directory. All metadata continues to push to the remote store as normal it is only when fetching a file that we would interface with the original snapshot if necessary. In a way its similar to a searchable snapshot dir however those code paths would not be reusable with the incoming writeable warmCompositeDirectoryimplementation. That dir handles block fetch aboveRemoteSegmentStoreDirectory.This would look something like below with new writes would push to the remote store as normal and reads flowing through FilteredRemoteDirectory.
A requirement here would be to enforce some level of deletion protection on the original snapshot for the lifetime of the index or at least until all segments from the original snapshot are merged away. We could do this with some new index level settings to validate at snapshot deletion time to ensure its not backing any existing index, similar to searchable snapshots.
Related component
Storage:Remote
Describe alternatives you've considered
Nothing - restore from snapshot as a hot index then migrate. Migrate data off cluster and somehow wire it up when the dir initializes. This is risky because remote store paths are determined at index creation.
Additional context
No response