Consider ingesters returning `ResourceExhausted` temporarily unavailable

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.

https://quickwit.io

Other

8.01k stars 327 forks source link

Consider ingesters returning `ResourceExhausted` temporarily unavailable #5155

Closed guilload closed 3 months ago

guilload commented 3 months ago

Description

Consider ingesters returning ResourceExhausted temporarily unavailable and report them as such, forcing the CP to open a new shard on another ingester. In addition, ingesters now reject InitShards requests when their WAL is full.

How was this PR tested?

Added and updated unit tests.

github-actions[bot] commented 3 months ago

On SSD:

Average search latency is 0.984x that of the reference (lower is better).
Ref run id: 2268, ref commit: 9fddb686f0024d58fc18dc77c44dc5358bbf0518
Link

On GCS:

Average search latency is 1.05x that of the reference (lower is better).
Ref run id: 2269, ref commit: 9fddb686f0024d58fc18dc77c44dc5358bbf0518
Link