Field Usage Analysis UI

cjcenizal commented 2 years ago

This relates to the Disk Usage Analysis UI (https://github.com/elastic/kibana/issues/101485). CC @giladgal @jpountz @dnhatn @elastic-jb

The problem

Per conversation with @lucabelluccini and @VimCommando, Elastic's adoption happy path helps users get started by enabling users to index data easily by applying default mappings. This becomes a problem years later as these users' deployments grow to be many TB in size, a large amount of which is consumed by unnecessary fields created by these default mappings. This translates into perceptions that Elastic is expensive and doesn't scale well.

The solution

We want to help users:

Identify the problem. Users should have a way of discovering that many fields aren't being used and are also contributing to costs.
Resolve the problem. Users need a tool and guidelines for removing unused fields.

Proposed UI

One idea we had was a UI that consists of a visualization of unused fields, driven by the Field usage API #73944, with a health-check option that can surface common issues and guide users to best practices.

Note that we still need to think through the flow for how a user would address their unused fields.

Unused fields visualization

Here's how Analyzer.Next visualizes the fields reported in a diagnostic. Note the number of text fields and keyword fields are almost the same in the first screenshot, indicating that many fields are using the default mapping of text/keyword pairs.

Health check

Some of the health check ideas we had were:

If the user only has a 1% difference (or some other threshold) between their keyword and text fields then they're probably not optimizing their mappings.
If logs are coming in with default mappings on them, then that’s probably also a poor optimization.
If an index is following the agent or beats default params, but it’s using the default mapping, then that could also be optimized.

Enhancements

We could also improve the UX by notifying users when they have indices that use default mappings.

We could ask ES to add metadata indicating if default mappings are in use, e.g. mappings-provided: true/false.
We could also link to the originating index template from each index. If we could also record index template version on the index itself, this would provide sufficient forensics to figure out when the mappings in a data stream changed, in case the index template gets changed at some point.

elasticmachine commented 2 years ago

Pinging @elastic/kibana-stack-management (Team:Stack Management)

giladgal commented 2 years ago

It would be good to combine in the UI the field usage API with the disk usage API (closed by #74051), so the user gets a clear view of the cost of the mapping decisions. It shows the two dimensions of the decisions: what you get from it and what you pay for it.

elasticmachine commented 2 weeks ago

Pinging @elastic/kibana-management (Team:Kibana Management)

elastic / kibana