search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.93k stars 24.74k forks source link

Integrate reactive_storage autoscaling with snapshot restore #98795

Open idegtiarenko opened 1 year ago

idegtiarenko commented 1 year ago

Description

At this moment elasticsearch is not scaling up (using reactive_storage autoscaling decider) when trying to restore indices that require more disk space then it is available.

Reactive storage decider is looking for unassigned shards that could not be allocated because DiskThresholdDecider is responding with NO decision due to the lack of space. Once those are detected and reported via autoscaling API, the orchestrator require additional time to actually resize existing or add new nodes.

On the other side snapshot restore operation adds shards to be restored to the routing table and makes a single attempt to allocate them. In such scenario shards would fail to allocate (as there no enough disk space) and will be marked as failed to restored almost immediately.

Once snapshot finishes (with a failure to restore big shards) reactive_storage decider is going to ignore the new unassigned shards as corresponding snapshot restore operation is already finished and they could no longer recover. This way cluster is not going to scale up afterwards (in case user wants to close those shards and retry snapshot restore).

This issue could be resolved if snapshot restore could allow multiple attempts to allocate shard when autoscaling is enabled.

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-distributed (Team:Distributed)