Cluster and node names with "-"

[andreacfm]

This is not the right place to ask .... I will try. From the Grafana labs page:

If your cluster_name or node names have "-" you will have to load a custom index to set the "name" field to not_analyzed

What do you exactly mean for loading a custom index? The issue I see is that the filtering by cluster name does not actually filter and all the data set is returned for each cluster selection. If I convert the queries of the panels as cluster_name.keyword:$Cluster the filtering works.

[trevorndodds]

It’s the way strings are analyzed.

You can upload a template to elasticsearch to prevent the field from being analyzed. But you can just use .keyword. The grafana template should be using .keyword.

More info... https://www.elastic.co/blog/strings-are-dead-long-live-strings

[morganwalker]

I think I'm having a similar issue. I'm using {"find": "terms", "field": "cluster_name.keyword"} to define $Cluster but no values are returned. I'm using Grafana 5.0.4 and ES 6.2 hosted with AWS. I'm not sure why the cluster name isn't returning.

[trevorndodds]

Does it show in Kibana under Discover?

[morganwalker]

I think you're asking about cluster_name and I don't see it in Kibana under Discover.

And to that point, it doesn't show up in /search-metrics?pretty, where search-metrics is my index.

[trevorndodds]

If you query the elasticserch directly http://elasticsearchURL:9200/_cluster/health?pretty does it show anything?

[morganwalker]

It does:

{
  "cluster_name" : "445227032534:monitoring-staging",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 7,
  "number_of_data_nodes" : 4,
  "active_primary_shards" : 11,
  "active_shards" : 22,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[trevorndodds]

Thanks, yeah that name is formatted differently than I would expect. I’ll take a look in the morning.

[stelcheck]

I tried to rename my one node cluster to something with just letters in the name, but still get the same issue:

http://elasticsearchURL:9200/_cluster/health?pretty

{
  "cluster_name" : "dawn",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "active_primary_shards" : 9,
  "active_shards" : 9,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 5,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 64.28571428571429
}

Addendum: The node name does still have "-" in it, could that be related (I doubt it, from what I read, but just to be sure)?

[trevorndodds]

@stelcheck yes but if you use name.keyword it should not matter, best to check in Kibana and Grafana that both "name" and "cluster_name" exist. then check below - under edit -> Preview of Values.

[trevorndodds]

@stelcheck I did a quick test, downloaded elasticsearch 6.3.1

  "cluster_name" : "monitoring-staging",
  "cluster_uuid" : "1ogFuDM1Rg6h_P4ZCk1-Qw",
  "version" : {
    "number" : "6.3.1",

Re-imported the dashboard to a new dashboard from https://grafana.com/api/dashboards/878/revisions/8/download

I had no problems with - in the clustername and node, it appeared right away. What elasticsearch version are you running?

@morganwalker I tried to set my cluster_name like yours but I get the below. Is this a managed elasticsearch from AWS or did you deploy it?

Exception in thread "main" java.nio.file.InvalidPathException: Illegal char <:> at index 12: 123451221316:monitoring-staging

As a test I update my cluster_name in the python script and it seems to display fine.

Maybe check you have the latest python and grafana dashboard?

[morganwalker]

@trevorndodds We needed alerting around these metrics as well so we ended up using a prometheus elasticsearch exporter and then building a graph on top of those prometheus metrics.

Thanks for the quick response and follow-ups.

[stelcheck]

@trevorndodds using ES 6.2.4. The variables are already set as you described, but I still get the same error. I also tried to switch to 6.3.1 as you did, to no avail.

I also tried to install the specific revision you linked to, but to no avail.

Update: Sincere apologies, it appears I was running the script incorrectly. Everything works fine now, please ignore me :)

[trevorndodds]

Glad it’s working :)

[Tanmoy3108]

Hi trevor,

I have an confusion we have to run the script in es server or grafana server. Please help as i am new to grafana and ES servers