[Discussion] OpenSearch searchable snapshot set-up in Glacier tier

[Sweetnesh]

Describe the bug

Hi knowledgeable folks of the forum,

Searching for some guidance and assistance with an OpenSearch searchable snapshot set-up. The intent here is to store the searchable snapshot in one of the cheaper Glacier tiers for cost effective (but still searchable) archival. Performance is not a major consideration here.

Done so far to:

Created snapshot in an S3 bucket. Restored snapshot using API : POST _snapshot///_restore with storage_type = remote_snapshot. Index is restored and tested searchable. Given the somewhat random nature of the data searches, cache has been minimizes as it would unlikely return any meaningful hits. A few open items so far:

When doing a search for results contained within the searchable index, will the search process all of the snapshot? Or is the search smart enough to jump directly to the matching blob units within the snapshot? Would greatly appreciate any thoughts or ideas how to pursue this further.

Thanks!

Related component

Search:Searchable Snapshots

To Reproduce

Created snapshot in an S3 bucket. Restored snapshot using API : POST _snapshot///_restore with storage_type = remote_snapshot. Index is restored and tested searchable.

Expected behavior

When doing a search for results contained within the searchable index, will the search process all of the snapshot? Or is the search smart enough to jump directly to the matching blob units within the snapshot?

Additional Details

No response

[mch2]

@sohami are u able to answer the above questions.

[sohami]

When doing a search for results contained within the searchable index, will the search process all of the snapshot? Or is the search smart enough to jump directly to the matching blob units within the snapshot? Would greatly appreciate any thoughts or ideas how to pursue this further.

@Sweetnesh It doesn't process all the snapshot but instead download blocks of data from blobs as needed by search request. The searchable snapshot index is powered by the RemoteSnapshotDirectory implementation which provides OnDemandBlockSnapshotIndexInput IndexInput for all the index files powering the searchable snapshot index. Ref here. This IndexInput impl has capability to download the blocks of the index file blobs as needed or accessed by the search request.

[sohami]

@Sweetnesh I am also curious if you configured the storage class in the s3 repository with Glacier or you are creating the snapshots with standard blobs and later moving those to glacier using s3 lifecycle mechanism ?

[Sweetnesh]

I am creating snapshot with standard blobs and then moving it to glacier using s3 lifecycle.

[Sweetnesh]

@sohami really appreciate the insights - feeling much more optimistic about this capability.

With a 'mounted' remote snapshot, if I understand correctly, only the index metadata is downloaded, and the data 'blobs' themselves are only accessed if required by the search query.

As far as I gather, using aggregate queries leverages the inverted indices created when OpenSearch ingests data, and does not access the data blobs.

So, would I be correct to expect the performance of aggregate queries run on remote mounted snapshots (with overall query size set to 0 to prevent any document fetching) to be quite snappy, since they rely solely on the index metadata downloaded at 'mount' time, and avoid fetching additional data from the remote storage repository?

[sohami]

With a 'mounted' remote snapshot, if I understand correctly, only the index metadata is downloaded, and the data 'blobs' themselves are only accessed if required by the search query

To clarify, when shards are being initialized, it opens the readers of each index files (or blobs in remote store) present in each segment of the shard. As part of reader open is when it downloads blocks from both metadata and data files to do some verification like checksum, etc. These minimum blocks which are downloaded is what is generally referred as the metadata in the context of searchable snapshot shards.

As far as I gather, using aggregate queries leverages the inverted indices created when OpenSearch ingests data, and does not access the data blobs.

Aggregate queries usually utilize the docValues index files (columnar index in lucene) and inverted index are more useful for term queries. So docValues or (.dvd) files will get accessed during the query execution.

So, would I be correct to expect the performance of aggregate queries run on remote mounted snapshots (with overall query size set to 0 to prevent any document fetching) to be quite snappy, since they rely solely on the index metadata downloaded at 'mount' time, and avoid fetching additional data from the remote storage repository?

Hope above one will clarify that there will still be data blocks downloaded for aggregation. First query could be slow but as your local cache gets warmed up with downloaded blocks, subsequent query can be faster if they don't need any new blocks to be downloaded. Also I am assuming when you say "data" blob, you mean data file for each index files on Lucene side.

[sohami]

@Sweetnesh Please feel free to close the issue if you don't have any more questions.