search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.73k stars 24.42k forks source link

Expose search-idle shard state in shard stats API #110688

Open DaveCTurner opened 2 weeks ago

DaveCTurner commented 2 weeks ago

A shard enters search-idle state if (a) index.refresh_interval is not set, and (b) there is an extended absence of search activity on the shard. When in this state it will not be refreshed, and this means that stats such as the doc count may appear to users be incorrect.

Today there's no direct way to observe that a shard is in search-idle state, although you can usually deduce it from other stats if you know for what to look. Still, this comes up in support cases from time to time. I think it'd be helpful to expose this state in the shard stats (within org.elasticsearch.index.refresh.RefreshStats makes sense to me) and possibly also in GET _cat/shards in order to help clarify when other stats exposed there may be out of date due to a lack of refreshes.

elasticsearchmachine commented 2 weeks ago

Pinging @elastic/es-search (Team:Search)

limotova commented 1 week ago

Hi, I was looking into this and it looks like this stat is actually being exposed when you set the level query parameter to shards, it just seems like none of these level-unique stats are getting documented? But the idle state also isn't currently being exposed in the CAT API (some of the other shard stats like dataPath are though). I think it makes sense to leave this stat in ShardStats, because RefreshStats also seem to be getting used to generate stats of multiple shards and I found some cases where I don't think it really makes sense to combine idle state But I can open a PR to add it to the CAT API..!

DaveCTurner commented 1 week ago

Aha thanks @limotova, I was looking at this on a 7.x cluster but you're right, we added these stats with https://github.com/elastic/elasticsearch/pull/95740 in 8.9.0.

I'm not too bothered about the _cat/shards column, but it would be good to address the missing docs.

limotova commented 1 week ago

I can try to take a shot at adding docs for stats that appear when you set the level parameter. Should I close the PR for the _cat API?

DaveCTurner commented 1 week ago

Should I close the PR for the _cat API?

Up to you, tho I suspect it's more work than it's worth to get it merged. I think the one test change you've made won't pass the BwC tests, and it needs other tests too.

limotova commented 1 week ago

I added an extra yaml test (with a questionable sleep I stole from another yaml test), but the ShardStats' isSearchIdle() method seems to be pretty thoroughly tested in SearchIdleTests so I'm not quite sure which other tests would need to be added?

elasticsearchmachine commented 1 week ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)