Transform response to not use "data" as JSON

[lukas-vlcek]

Several ES admin REST endpoints return JSON response that uses "data" as hash keys. Example of this is cluster health API that returns indices section like this:

{
  "indices": {
    "index_01": { "number_of_shards": 5, "number_of_replicas": 3 },
    "index_02": { "number_of_shards": 4, "number_of_replicas": 2 }
  }
}

This typically occurs in all cases where the nodes, indices, shards or segments are part of the response. This can be problematic, for example, if the data are later stored into Elasticsearch then this causes several problems:

• With every new name (index, shard, node, segment, ...) the Elasticsearch mapping is modified. This can be a major issue.
• It is harder to construct useful queries and aggregations.

Introduce new watches option to modify the response data to mitigate/fix issues described above.

---

Prior art: we were doing the same thing in old elasticsearch-river-sysinfo.

[richm]

and then you will have trouble to visualize in Kibana . . . https://github.com/elastic/kibana/issues/1084

[lukas-vlcek]

Quickly reading through the thread... on the data side I do not see other alternative than using nested objects. For the UI layer, it is good to know Kibana is no-go. We will simply use other ways of visualizing this data.

[richm]

. . . or see if Kibana will eventually support that feature

[lukas-vlcek]

@richm I recommend you to read the thread then. It is a long reading but valuable. (Note: the GitHub UI collapsed some of the posts in the thread - maybe because the thread is long, I recommend you to expand even this part and read all the stuff...)

[lukas-vlcek]

Seems that there is some effort to support nested aggregations in Grafana, but still it is unclear when this feature will be merged, see https://github.com/grafana/grafana/pull/4694

[lukas-vlcek]

Just for the reference, as of writing the latest version of Kibana community contribution to support nested aggregations can be found here: https://github.com/homeaway/kibana/commits/nestedSupport-5.2

It also makes sense to add reference to recent Kibana presentation from Elastic{on} 2017 https://www.elastic.co/elasticon/conf/2017/sf/what-is-cookin-in-kibana

• I did not notice support for nested aggregations is on the published road map
• Important internal changes are coming:
  • Creating contracts between Kibana and its plugins (slide 45)
  • Switching from Angular.js to React.js (slide 46)

[lukas-vlcek]

Related: https://discuss.elastic.co/t/elasticsearch-query-in-kibana/81467/2

Unfortunately, Kibana doesn't support nested aggregations. You don't be able to use the query you've created in Kibana.