elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
70k stars 24.76k forks source link

Shrink an index from a snapshot #73500

Closed dakrone closed 2 years ago

dakrone commented 3 years ago

In order to help address DTS costs, and also reduce the complexity of ILM's shrink operation, we should investigate adding the ability to shrink an index from a snapshot rather than from local disk.

It would be nice if instead of shards having to be colocated, we could take a snapshot of the index, and then perform a restore and shrink at the same time. This would not only alleviate https://github.com/elastic/elasticsearch/issues/63519 but it would also be much less error prone to issues like a situation where not all the shards can fit on a node.

In order to accomplish this, we would need to introduce the Default Repository (#66040), so that a repository does not need to be specified for the shrink request.

Note that this diagram shows the combination of shrink and forcemerge as described in #73499, but this is not a prerequisite, as the two are independent.

3B7F1C0B-B8E8-4E7D-A871-76D117580208

One extremely beneficial side effect of implementing shrink like this is that shrink would no longer need to perform the slightly error-prone behavior of identifying a single node to move shards prior to the shrink (which itself can cost users money depending on where their data resides). It also would remove some code complexity on the ES side for handling shrink behavior during cluster/node shutdowns.

elasticmachine commented 3 years ago

Pinging @elastic/es-distributed (Team:Distributed)

tlrx commented 2 years ago

We discussed this in team today and we think that snapshot based recoveries reduced most of the DTS costs while the increasing usage of datastreams reduced the need to shrink indices. We also haven't seen much shrinking issues that would justify the significant effort required to build this, so I'm going to close this.