search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
1.07k stars 24.83k forks source link

Field caps - Ability to exclude fields by name #99450

Open mattkime opened 1 year ago

mattkime commented 1 year ago

When mapping explosions occur, they're often something like node_id_12345. Unfortunately we don't have a way to exclude these for any field request even though they make the mapping much too large and are best ignored.

The field caps api should support negating wildcards such as -node_id*. This should speed up worst case field loading when mapping explosions are present.

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-search (Team:Search)

sophiec20 commented 1 year ago

We have seen multiple examples where there are 100s of fields which are not relevant for users to see. These extra fields have snuck in due to legacy mapping problems, or ingest errors, or poor schema choices, or bloat in agent collection. Sometimes it just depends on the context. The problem is exacerbated by wide index patterns e.g. logs-* which are common in solutions.

These extra fields create noise and are bad for the user experience in Kibana. They appear in search results, Discover field lists, in data view management, in KQL autocomplete, in Lens builder, in field selection drop downs, etc. Searching for useful fields becomes problematic when there are many useless fields. Having a single unified way in elasticsearch to easily exclude/include the same mappings from searches and from field enumeration would be useful so it can be applied consistently.

At enterprise scale, fixing this via reindex is not a viable option.

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-search-foundations (Team:Search Foundations)