GET field count per index API

[Leaf-Lin]

Today, there's an API to get per index field or all fields:

GET <index>/_field_caps?fields=*

which returns:

{
  "indices" : [
    "my_index",
    "my_index2",
    "my_index3",
    ...
  ],
  "fields" : {
    "my_field1" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    "my_field2" : {
      "keyword" : {
        "type" : "keyword",
        "searchable" : true,
        "aggregatable" : true
      }
    },
    ...
  }
}

But the output from the above doesn't have break down by index, so it's not easy to troubleshoot which index is having mapping explosion problem.

I can potentially loop over my indices list and running GET <index>/field_caps?fields=* API to extract the field length per index, but again, this is not ideal.

It would be most useful if Elasticsearch can have an API out-of-the-box to count the number of fields breakdown by the indices, expected output should look like:

index_name      field_count
my_index        10
my_index2       20
my_index3       30

This can be part of the GET indices/stats or cat indices API.

Btw, kibana might be able to take advantage of the API instead of doing its own aggregation/counting to show index pattern field count:

---

Not sure if the following is the best script to extract the field count, but with some jq over the GET _mapping output, I am able to get the desired format:

curl -XGET https://localhost:9200/_mapping > mapping.json
jq -r '[to_entries[]| .key as $index|  [ .value.mappings|to_entries[]|{(.key): ([.value|..|.type?|select(.!=null)]|length)} ]|map(to_entries)| flatten| sort_by(.value)|from_entries as $types | {index_name: $index, index_field_count: ([$types|to_entries[].value]|add)}]|.[]|.|map(.)|@tsv ' mapping.json  | column -t  | sort 

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[mbrunnert]

This feature is becoming more valuable now that the documentation recommends sizing heap based on field counts

[geekpete]

Seems we have field counts with Field Usage Stats api, but it's not so consumable with some top level counts.

GET /*/_field_usage_stats

Output is per shard and recursive, so getting totals is still a manual exercise. Potential for a feature request to add some extended parameter to do recursive field counts and show totals at each level.

And it's only doing actively used fields not all fields in the mapping? So that could be something that extended could also do.

[Hythloday-zero]

https://www.elastic.co/guide/en/kibana/master/data-views-api-get.html#data-views-api-get works to get the raw info similar to the Kibana UI screenshot, but it doesn't get us a fields.count value which would be extremely valuable.

[shaigbdb]

Seems we have field counts with Field Usage Stats api, but it's not so consumable with some top level counts.

GET /*/_field_usage_stats

This returns field usage information - I'm not seeing a field count in the output.

[geekpete]

Sorry I meant the response returns the lists of fields but counting them is something you need to do manually.

[inqueue]

Bump. Is there any chance of getting this in the near term? Elasticsearch has already imposed a field limit via index.mapping.total_fields.limit for quite some time and having field count information available would be very useful for understanding the usage, especially for dynamically mapped indices. One shouldn't have to be a JQ 🥷 or require Kibana to get the index field count!

[lucabelluccini]

Especially following the introduction of index.mapping.total_fields.ignore_dynamic_beyond_limit via https://github.com/elastic/elasticsearch/pull/96235 (and it being used in 8.14+ on several index templates of Fleet), it is crucial to get the field count at runtime in order to eventually automate / warn if the field count is reaching the limit.

A poor's man way of counting fields might be gron mappings of index | grep '.type =' -c, but it seems Elasticsearch doesn't only count "leaf" fields but also the parent field as object.

In addition, the count reported by a Kibana Data View is incorrect:

As if I try to add few fields to the index above, I get a rejection due to field limit exceeded (10000).

 (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:3999] failed to parse: Limit of total fields [10000] has been exceeded while adding new fields [19]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [10000] has been exceeded while adding new fields [19]\"}},

FYI @flash1293 / @felixbarny

[felixbarny]

I agree that this would be very useful. I feel like we're re-implementing this on the client side in multiple places (and not always in the correct way). For example, in the dataset quality page. cc @achyutjhunjhunwala @gbamparop.

I think exposing this in some ES API would be relatively straightforward as we already have a method for this in MappingLookup:

https://github.com/elastic/elasticsearch/blob/8db918110c377830ff17da49e980be8dc0e9524e/server/src/main/java/org/elasticsearch/index/mapper/MappingLookup.java#L250-L255

[VimCommando]

On a related note I would love to see field counts, and breakdown by field type, in the <index>/_stats API. It would make alerting for mapping explosions trivial.

[felixbarny]

I've created a quick POC for adding the fields count to the index stats API: https://github.com/elastic/elasticsearch/pull/116438

[elasticsearchmachine]

Pinging @elastic/es-data-management (Team:Data Management)

[felixbarny]

I discussed this with @dakrone. A challenge with adding this to the index stats API is that this API isn't available on serverless, as it's considered to be exposing too many low-level details. However, for this use case we'll want to have a user-accessible API.

We discussed adding this to get mappings. It has some technical challenges as get mappings is answered by the coordinating node, getting the unparsed mapping from cluster state. To call MappingLookup#getTotalFieldsCount, we would either need to delegate to the primary shard, or parse the mapping on the coordinating node. It'll also be a bit awkward to return a property for the field count from get mappings that's not allowed to be present in a put mappings request. It may break use cases where a client gets the mapping, modifies it, and updates it. Only adding the field count in the get mapping response when a verbose flag is enabled could mitigate the breaking aspect of it. The awkwardness is still there.

[javanna]

I can see how having clear field count per index is valuable, especially as this is reverse-engineered now in multiple places, and probably not aligned with the ES logic necessarily. I see the challenges with using index stats API, as well as get mappings. It sounds like this info could be added to field caps more easily, although it does not map strictly to the capabilities of fields.

[felixbarny]

I forgot to mention that we also discussed adding this to field caps. A challenge with that approach is that field caps isn't focused on a particular index, but at an index pattern. Even if you're looking at just a single data stream, it has multiple backing indices, where each may have a different number of fields. While we could add another section in the field_caps response that lists the number of fields for each matching index, it seems to go against the spirit of that API which aggregates data for fields across multiple indices. For this, we probably want an API that's focussed on indices rather than fields.

Is the cat indices API available on serverless?

[dakrone]

Is the cat indices API available on serverless?

Yes, cat indices is available for Serverless (though it should be a human-invoked thing, and not used or relied upon programmatically).

[felixbarny]

The index stats still seems like the most appropriate place to add this. It's index-centric and already fetches stats from the shards rather than just answering on the coordinating node, which makes accessing MappingLookup#getTotalFieldsCount straightforward. The only downside is that it doesn't work for serverless. But we could think about allowing the index stats api for serverless but limit the index metrics that we're exposing. But I'm not sure if there's precedent for partially exposing an API in serverless.