[Dashboards] Enable easy filtering by data tier

[thomasneirynck]

Describe the feature:

Allow filtering by data tier. There are several use-cases where this would be beneficial.

• edit mode for visualizations (e.g. in Lens) only filters on hot tiers
• allow dashboard users to opt-out of slow searches on cold tiers

Describe a specific use case for the feature:

Consider a top level toggle in the Dashboard, which allows users to filter out tiers. In the background, all queries would get a filter applied like:

"query": {
    "terms": {
      "_tier": ["data_hot", "data_warm"] 
    }
  }

This does not apply just to querying data. It is also relevant when querying field-lists for dataviews (https://github.com/elastic/kibana/issues/139340).

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

@thomasneirynck is this still relevant?

[thomasneirynck]

I'll remove the data-view tag.

wrt Dashboards, it's still open. @teresaalvarezsoler wdyt?

[lukasolson]

Just wanted to add here that it's possible to do this in KQL (although there isn't any autocomplete support for this): _tier: (data_hot or data_warm)

[teresaalvarezsoler]

edit mode for visualizations (e.g. in Lens) only filters on hot tiers

Not sure this is adding much value at authoring time because there is an easier (and more clear) workaround which is selecting a smaller timeframe

allow dashboard users to opt-out of slow searches on cold tiers

This is where I can see most of the value, but I think we need to discuss this a bit further, some concerns:

• It exposes an admin setting to end-users. I'm not sure how many of them even know what data tiers mean.
• Even if they are aware, is the decision to exclude some tiers too complex? i.e., how would they know "how much" data they are filtering out? How would they know if they can even trust their results?

I think if we decide to do it we should only suggest this setting when 1) there are different data tiers configured 2) there is a query taking too much time that is querying a cold and/or warm tier and a hot tier. In this situation we can show a modal suggesting users to filter out the cold/warm tiers. I think we should get some data first to figure out how frequent this situation is.

[sophiec20]

if we do this, suggest "must_not": { "terms": { "_tier": [ "data_frozen", "data_cold" ] } }