search

quickwit-oss / quickwit

Cloud-native search engine for observability. An open-source alternative to Datadog, Elasticsearch, Loki, and Tempo.
https://quickwit.io
Other
8.01k stars 327 forks source link

Searcher fails to answer probe fast enough. #4885

Open fulmicoton opened 5 months ago

fulmicoton commented 5 months ago

As identified on airmail...

(It could be the disk cache loading. We probably need to have two level startup like we do for ingesters.)

fulmicoton commented 5 months ago

Right now we have a readiness and a liveness probe. We also do not have any startup probe.

The readiness probe relies on what is advertise on the cluster. The liveness probe checks the indexing service and the janitor if any, by ask them an adhoc message called Healthz.

A node readiness reporting tasks polls a bunch of conditions and updates the READINESS key of the self cluster state.