search

opensearch-project / OpenSearch

🔎 Open source distributed and RESTful search engine.
https://opensearch.org/docs/latest/opensearch/index/
Apache License 2.0
8.89k stars 1.63k forks source link

[Writable Warm] Recovery/Replication flow for Composite Directory #13647

Open rayshrey opened 2 weeks ago

rayshrey commented 2 weeks ago

Is your feature request related to a problem? Please describe

With the introduction of Composite Directory for Writable Warm, we will also need to modify it's replication/recovery flow since we only have partial data available locally in composite directory.

Describe the solution you'd like

Replication/Recovery flow should work for Composite Directory. Will update this section with more details once have a concrete LLD for this.

Related component

Storage:Remote

Describe alternatives you've considered

N/A

Additional context

Writable Warm RFC - https://github.com/opensearch-project/OpenSearch/issues/12809 Composite Directory RFC - https://github.com/opensearch-project/OpenSearch/issues/12781

mch2 commented 2 weeks ago

A couple of things come to my head here on how this would be inefficient but I'm not sure if it would actually break.

We wouldn't want to download all files from the remote store given its a warm case. There are a couple of locations where this happens today - prior to recovery start, prior to engine open, and on an engine reset / failover case.

Second I think prior to engine open we fetch the latest SegmentInfos in memory and perform a local commit with SegmentInfos.commit so that the engine can start from reading the latest segments_n in the directory. I think this would still work but may be downloading more than we want to perform the commit. If thats the case we may need to update this flow to ensure the remote store has a commit file and load that.

nisgoel-amazon commented 1 week ago

assign it to me