Node Indices Status API is significantly slower

[loe]

curl http://localhost:9200/_cluster/nodes/_local/stats?all=true used to take a few milliseconds against my development installation (0.90.2 on OS X). I upgraded to 1.0.0 and adjusted my URL, and now it takes anywhere from 2-4 full seconds.

13:28:38 ~ → time curl http://localhost:9200/_nodes/_local/stats/indices
{"cluster_name":"elasticsearch_andrew","nodes":{"6QXHJw0uTAe7GRKo073LOg":{"timestamp":1392931719502,"name":"Saturnyne","transport_address":"inet[/127.0.0.1:9300]","host":"jester.local","ip":["inet[/127.0.0.1:9300]","NONE"],"indices":{"docs":{"count":1258,"deleted":221},"store":{"size_in_bytes":795157,"throttle_time_in_millis":0},"indexing":{"index_total":2,"index_time_in_millis":63,"index_current":0,"delete_total":30,"delete_time_in_millis":106,"delete_current":0},"get":{"total":0,"time_in_millis":0,"exists_total":0,"exists_time_in_millis":0,"missing_total":0,"missing_time_in_millis":0,"current":0},"search":{"open_contexts":0,"query_total":0,"query_time_in_millis":0,"query_current":0,"fetch_total":0,"fetch_time_in_millis":0,"fetch_current":0},"merges":{"current":0,"current_docs":0,"current_size_in_bytes":0,"total":0,"total_time_in_millis":0,"total_docs":0,"total_size_in_bytes":0},"refresh":{"total":56,"total_time_in_millis":105},"flush":{"total":10,"total_time_in_millis":352},"warmer":{"current":0,"total":73,"total_time_in_millis":1},"filter_cache":{"memory_size_in_bytes":0,"evictions":0},"id_cache":{"memory_size_in_bytes":0},"fielddata":{"memory_size_in_bytes":0,"evictions":0},"percolate":{"total":0,"time_in_millis":0,"current":0,"memory_size_in_bytes":0,"memory_size":"0b","queries":0},"completion":{"size_in_bytes":0},"segments":{"count":121,"memory_in_bytes":324127668},"translog":{"operations":0,"size_in_bytes":0}}}}}
real    0m2.211s
user    0m0.005s
sys 0m0.004s

On production instances I scrape these stats with python and collectd to monitor my cluster.

[loe]

I should add my machine is quite fast. Core i7, 8GB RAM, SSDs. It is only the indices stats that are slow, all others (java, os, process, http etc. all return in a few milliseconds as before).

[s1monw]

this might be related to https://github.com/elasticsearch/elasticsearch/issues/5201 ?

[s1monw]

is it possible that you have some old Lucene 3 segments in your index that were created with 0.20?

[kimchy]

FYI: the segments API will give back the Lucene version the segment was created with

[loe]

I do indeed have a variety of different "versions", 3.6, 3.6.2 and 4.3.

Is there a way to upgrade these?

[s1monw]

@loe you can upgrade your segments which usually happens during a merge. If you optimize you index it will be the latest format. Yet this might take time and might even have an impact on your cluster while you run it so be careful with this - it really depends on how much data is in there though.

[loe]

So I ran optimize against my indexes (curl -XPOST http://localhost:9200/_all/_optimize) but I still have some 3.6.2 segments in my shards. Do I need to have more churn in order to trip merges?

[s1monw]

hmm this is odd - what did the optimize call return?

[loe]

total":36,"successful":36,"failed":0}}

[s1monw]

hey @loe my assumption is that there is really just one segment and that means it will not re-write the segment. I don't think you have a chance to do that at this point.

[clintongormley]

The optimize call now supports a force option #5293