GET field count per index API

[Leaf-Lin]

Today, there's an API to get per index field or all fields:

GET <index>/_field_caps?fields=*

which returns:

{
  "indices" : [
    "my_index",
    "my_index2",
    "my_index3",
    ...
  ],
  "fields" : {
    "my_field1" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "my_field2" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    ...
  }
}

But the output from the above doesn't have break down by index, so it's not easy to troubleshoot which index is having mapping explosion problem.

I can potentially loop over my indices list and running GET <index>/field_caps?fields=* API to extract the field length per index, but again, this is not ideal.

It would be most useful if Elasticsearch can have an API out-of-the-box to count the number of fields breakdown by the indices, expected output should look like:

index_name      field_count
my_index        10
my_index2       20
my_index3       30

This can be part of the GET indices/stats or cat indices API.

Btw, kibana might be able to take advantage of the API instead of doing its own aggregation/counting to show index pattern field count:

---

Not sure if the following is the best script to extract the field count, but with some jq over the GET _mapping output, I am able to get the desired format:

curl -XGET https://localhost:9200/_mapping > mapping.json
jq -r '[to_entries[]| .key as $index|  [ .value.mappings|to_entries[]|{(.key): ([.value|..|.type?|select(.!=null)]|length)} ]|map(to_entries)| flatten| sort_by(.value)|from_entries as $types | {index_name: $index, index_field_count: ([$types|to_entries[].value]|add)}]|.[]|.|map(.)|@tsv ' mapping.json  | column -t  | sort 

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[mbrunnert]

This feature is becoming more valuable now that the documentation recommends sizing heap based on field counts

[geekpete]

Seems we have field counts with Field Usage Stats api, but it's not so consumable with some top level counts.

GET /*/_field_usage_stats

Output is per shard and recursive, so getting totals is still a manual exercise. Potential for a feature request to add some extended parameter to do recursive field counts and show totals at each level.

And it's only doing actively used fields not all fields in the mapping? So that could be something that extended could also do.

[Hythloday-zero]

https://www.elastic.co/guide/en/kibana/master/data-views-api-get.html#data-views-api-get works to get the raw info similar to the Kibana UI screenshot, but it doesn't get us a fields.count value which would be extremely valuable.

[shaigbdb]

Seems we have field counts with Field Usage Stats api, but it's not so consumable with some top level counts.

GET /*/_field_usage_stats

This returns field usage information - I'm not seeing a field count in the output.

[geekpete]

Sorry I meant the response returns the lists of fields but counting them is something you need to do manually.

[inqueue]

Bump. Is there any chance of getting this in the near term? Elasticsearch has already imposed a field limit via index.mapping.total_fields.limit for quite some time and having field count information available would be very useful for understanding the usage, especially for dynamically mapped indices. One shouldn't have to be a JQ 🥷 or require Kibana to get the index field count!