Pull down for selecting Nodes works only for the first 10 items

[joealex]

I have an ES cluster with "n" nodes. The script is loading stats correctly in to the elasticsearch-metrics index for all the "n" Nodes. When viewing the dashboard, Cluster level dashboards are populated fine for any selection. If you select All or individually a Node only the first 10 Nodes in the list are displaying data in the Node level panels. The dropdown shows all the "n" nodes correctly. Did try adding "size" : n to the template but no effect.

[joealex]

Attaching screenshots. Notice the First 10 Nodes in dropdown when selected individually or All shows Node level data. When the 11th or 12th or others selected the Node panels dont show any data. See Kibana screenshot which shows data is there in the index for those also.

[trevorndodds]

That is odd, Thanks I'll take a look.

[joealex]

I am on ES 6.0.0-beta2 shouldn't make much diff than 5.5, but FYI

[joealex]

I did turn up the log level to debug but not too much info there. Here is what I see.

When initially opens dashboard this is the query t=2017-09-14T16:34:01+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/_msearch status=200 remote_addr=::1 time_ms=12 size=5057 referer="http://localhost:3000/dashboard/db/elasticsearch-dashboard?refresh=10s&orgId=1"

Dont see any Node names or All in the query but data for the first 10 Nodes in the dropdown list is shown when Node panels are opened. If a panel is open the changes in dropdown does not reflect in the log query but either way data will always be pulled for first 10 Nodes. On opening a Node panel the query in the log starts to reflect the selected Nodes or All, but not sure if the real query is same which goes to ES.

Is this in anyway related to ES by default returns 10 results. I have added the size parameter in template anyway.

The query shows the no data Nodes and also the first 10 Nodes or All, but reflected only when you open a Node panel.

t=2017-09-14T16:31:46+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/_msearch status=200 remote_addr=::1 time_ms=32 size=15357 referer="http://localhost:3000/dashboard/db/elasticsearch-dashboard?refresh=10s&orgId=1&var-Cluster=my_cluster&var-Node=All&var-nodename=name.keyword"

t=2017-09-14T16:31:26+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/_msearch status=200 remote_addr=::1 time_ms=20 size=5793 referer="http://localhost:3000/dashboard/db/elasticsearch-dashboard?refresh=10s&orgId=1&var-Cluster=my_cluster&var-Node=10p31&var-nodename=name.keyword"

t=2017-09-14T16:33:19+0000 lvl=info msg="Request Completed" logger=context userId=1 orgId=1 uname=admin method=POST path=/_msearch status=200 remote_addr=::1 time_ms=17 size=8013 referer="http://localhost:3000/dashboard/db/elasticsearch-dashboard?refresh=10s&orgId=1&var-Cluster=my_cluster&var-Node=11p12&var-Node=11p13&var-nodename=name.keyword"

[joealex]

Do see an entry in deprecation log as below FYI

[2017-09-14T17:33:35,607][WARN ][o.e.d.s.a.InternalOrder$Parser] Deprecated aggregation order key [_term] used, replaced by [_key]

[joealex]

When tried selecting nothing in dropdown saw the following in ES logs. That must be since name is empty name(). But what was interesting was the size:10 in the aggregations 8 query is that the issue here ?

source={ "size" : 0, "query" : { "bool" : { "filter" : [ { "range" : { "@timestamp" : { "from" : "1505395866792", "to" : "1505406666792", "include_lower" : true, "include_upper" : true, "format" : "epoch_millis", "boost" : 1.0 } } }, { "query_string" : { "query" : "name:()", "fields" : [ ], "type" : "best_fields", "default_operator" : "or", "max_determinized_states" : 10000, "enable_position_increments" : true, "fuzziness" : "AUTO", "fuzzy_prefix_length" : 0, "fuzzy_max_expansions" : 50, "phrase_slop" : 0, "analyze_wildcard" : true, "escape" : false, "boost" : 1.0 } } ], "adjust_pure_negative" : true, "boost" : 1.0 } }, "aggregations" : { "8" : { "terms" : { "field" : "name.keyword", "size" : 10, "min_doc_count" : 0, "shard_min_doc_count" : 0, "show_term_doc_count_error" : false, "order" : { "_key" : "asc" } }, "aggregations" : { "2" : { "date_histogram" : { "field" : "@timestamp", "format" : "epoch_millis", "interval" : "1m", "offset" : 0, "order" : { "_key" : "asc" }, "keyed" : false, "min_doc_count" : 0, "extended_bounds" : { "min" : "1505395866792", "max" : "1505406666792" } }, "aggregations" : { "1" : { "max" : { "field" : "indices.segments.doc_values_memory_in_bytes" } }, "3" : { "max" : { "field" : "indices.segments.index_writer_memory_in_bytes" } }, "4" : { "max" : { "field" : "indices.segments.memory_in_bytes" } }, "5" : { "max" : { "field" : "indices.segments.stored_fields_memory_in_bytes" } }, "6" : { "max" : { "field" : "indices.segments.terms_memory_in_bytes" } }, "7" : { "max" : { "field" : "indices.segments.version_map_memory_in_bytes" } } } } } } } }}

[trevorndodds]

Yeah I suspect the issue is on the grafana dashboard side, some limit there. I haven't had time to look yet.

[trevorndodds]

Yeah the issue is that the Node specific graphs are limited to 10. If you select the one node not displaying and then F5 the page it will display the graphs. The one workaround is to remove the limit of 10 to No Limit.

[trevorndodds]

I'll remove the Limit and update for now. This could be a grafana bug.

[joealex]

In fact that was the issue. In the elasticsearch-dashboard I replaced the "size": "10" with "size": "0" and everything works perfectly

[joealex]

Summary - This is what worked for me. In the dashboard json update the size to 0.

In ES 6.0 only there is a minor deprecated item - more of annoying constant log entries. For this in the dashboard json update "orderBy": "_term" with "orderBy": "_key"

Thanks Trevor for all the help, excellent dashboard for ES metrics

[trevorndodds]

Great, glad to hear!