Commands for indices and nodes stats

[lukas-vlcek]

In order to allow easy collection (and interpretation) of operational statistics at indices and nodes level we need to consider adding two new extra commands.

The motivation is to have command(s) that can output all indices or nodes stats without additional need to alter the data before indexing it and at the same time it should be easy to query and pull & display the data in Kibana (so all the limitations of Kibana need to be considered upfront).

Details

Nodes stats API returns data in the form:

curl http://localhost:9200/_stats?level=indices

{
  "_shards": { ... },
  "_all": {
      "primaries": { ... },
      "total": { ... }
  },
  "indices": {
      "index1": {
          "primaries": { ... },
          "total": { ... }
      },
      "index2": {
          "primaries": { ... },
          "total": { ... }
      },
    ...
  }
}

We want to be able to collect all indices stats into common ES index. This means that from the above JSON response we want to pull all indices from .indices, add timestamp field into each and store it as a separate document. We should also consider including the ._all object.

In fact we will transform the JSON into nested form first.

Indices stats API returns data in the form:

curl http://localhost:9200/_nodes/stats

{
  "cluster_name": "...",
  "nodes": {
      "node1": { ... },
      "node2": { ... },
      ...
  }
}

Here we again want to pull all individual node info from the response and push it into common ES index as separate documents each containing the timestamp.

Notice, it is possible to get information about individual indices on each node using: curl http://localhost:9200/_nodes/stats?level=indices but at this point it is hard to imagine how we could effectively use and represent such data in Kibana dashboard.