[Controls] Individually Pre-Filter Available Results

[ThomThomson]

Feature Request

Controls currently are filtered by the Dashboard Context, and by selections made in previous controls. There is no other way for a dashboard author to narrow down the available options.

There are many use-cases in which a field has many values, but only a certain subset of those values are relevant to the dashboard.

Fix

In the editor for Data Controls, we should add a selection that allows authors to Narrow down available selections. This should take a form similar to a compressed unified search bar where an author can use filter pills and/or a KQL query bar to specify one or more filters which are always applied to search queries run by the control. These should be appended to any filters added by the dashboard or by previous controls.

Needs design @andreadelrio

[elasticmachine]

Pinging @elastic/kibana-presentation (Team:Presentation)

[ThomThomson]

It's important to note that there is a workaround to this issue raised by @llermaly, in which a pre-filtered alias is used to fetch the values for the control.

[shubhu934]

One of our client has requested this feature where the the problem is not that it doesn’t filter, the problem is that it shows all the values, what they would like, just as filters can be applied to visualizations, also apply them to these filters

[MakoWish]

I would really like to see the controls filterable on the back-end. The "workaround" is not really a solution, because that requires very granular Data Views for every control we want to use, and that is just going to make the environment very convoluted. The "deprecated" controls functioned much better in that they could be filtered on the back-end (although the filtering did not work), and they can be resized and placed anywhere on the dashboard. The new controls are very clunky. My vote is to fix the old controls and return them to GA.

[ThomThomson]

@MakoWish, you're correct that the workaround is quite clunky. We're currently prioritizing this functionality for an upcoming release.

We're also working on defining a project to allow Controls to be placed / resized freely on a Dashboard, and I will link the issue here once the project is defined and the issue is created. Hopefully these two fixes will make the new Controls suitable for your use case.

[MakoWish]

Any reason the "deprecated" controls aren't just fixed? If the filtering bug can be fixed on the List Control, it just seems it would be easier to fix those and return them to GA instead of completely reinventing the wheel. It just seems like a waste of time to me to put a bunch of effort into making the new controls function like the old ones.

[ThomThomson]

Good question @MakoWish. We made the decision to build a new Controls framework due to many fundamental architectural problems with the legacy Controls. These underlying problems were why the legacy controls were never made GA in the first place.

One of the major problems they had, and the most relevant to this discussion was the inability to be affected by the unified search bar at all. This was not a trivial bug, but a fundamental UX / code structure problem. In short, because the legacy controls published all of their selections as filter pills, having filter pills affect them in return would cause a loop. Because of this, the old controls were never affected by the unified search bar.

[P1llus]

I want to add some comment to this after discussing this internally.

Most integration dashboards (at least newer ones) do not have dashboard level filters, which usually could filter the new Controls I believe, so when you have 3 integrations installed, and you view Dashboard 1, and you see options from Integration 2 in your Dashboard 1 Controls, its confusing.

Most Controls are often on ECS fields, so they have a high chance of conflicts.

We very widely use visualization based filters rather than dashboard wide filters, because they are more hidden and cannot be deleted by mistake, which is a bad combination with the new Controls which currently do not have a filter.

It would be great if the "Create Controls" UI also had a field for a filter, like KQL etc.

[Erikg346]

Hello, is there an update on this?

[teresaalvarezsoler]

Hi @Erikg346, unfortunately, we are not planning to do this in the short-term for our current controls. However, we have a project in the mid-term where this will be possible for charts built using ES|QL queries. Please feel free to provide any details for your use case, it will help us build a solution that works for all users.

[Erikg346]

Hey @teresaalvarezsoler Our use case is the same as the ones listed in this thread, as well as the mentioned issues. For example, we might have a custom dashboard to monitor a cluster of SQL Servers. Using metrics—*, we might want to add filters in Controls to narrow it down to a certain number of host names (using host.name). By adding the 'pre-filter,' it can show the most relevant options for that specific dashboard.

This is having been one of the main reasons why we haven't fully migrated off of legacy Controls.

[MakoWish]

@teresaalvarezsoler,

The filtering on the legacy Controls is unfortunately broken (it never worked that I recall), but that was the way to go. I have been begging a pleading for years for those to be fixed, but I keep being told the new controls are the focus, and there will be no effort to fix the legacy ones. The entire point behind the controls is to pre-filter a dashboard, but if you cannot pre-filter the results shown in Controls, it really defeats the entire purpose.

[andreadelrio]

Hi folks, we're currently starting the design process for this feature's UI and thought we could gather some input here, given the interest shown so far. cc @MakoWish @Erikg346

Questions

• Would the KQL/Lucene bar be enough for your needs? or do you think filter pills are also required?

• Is there anything else you would like to have available? For example, would you like to be able to load a saved query for this?

[Erikg346]

Filter Pills are all I need. I would say an alternative would be to keep the same design as Lens:

[MakoWish]

If both cannot/will not be included, the "filter pills" are preferred IMO, and allowing the use of a Saved Search (not Saved Query - those are useless), would be ideal.

Side note: The reason I say Saved Queries are useless is because once they are defined, the values are statically assigned to any visualizations they were applied to. Updating the Saved Query later has zero effect on any visualizations that were based on that Saved Query. The visualizations will retain the original Saved Search values. Sorry to be blunt on those, but I have to reiterate they are completely pointless IMO. A Saved Search is the way to go, because any visualizations they were applied to (only available on the "Legacy" visualizations, not Lens or TSVB) are automatically updated any time there are changes to the Saved Search. Allowing a Saved Search to be applied to Lens has also been something I have been asking for since Lens was introduced.

[andreadelrio]

Sharing proposed designs for development. Figma can be found here.

[MedRS-IA]

I'm facing an issue with the new version of controls in Kibana, which limits the flexibility of positioning these controls within dashboards. In my current dashboards, it’s essential to place controls close to the specific visualizations they apply to, ensuring a logical and user-friendly experience. However, the new version of controls does not allow free positioning, which creates a challenge in cases where filters need to apply to certain visualizations only.

Could you please let me know if there are plans to enable flexible positioning of controls, similar to other components within Kibana dashboards? Would this feature be considered in an upcoming minor or major update?

Thank you for your work and for considering user feedback.

[teresaalvarezsoler]

Hi @MedRS-IA thanks for your request. There are not short-term plans to make controls as panels, but we will be revisiting this decision soon in the context of a related project. You can follow up updates here https://github.com/elastic/kibana/issues/154749

it’s essential to place controls close to the specific visualizations they apply to Can you clarify this in the meantime? Controls always apply to all panels in a dashboard, i.e., you cannot choose which visualizations they apply to so I'm not sure how you are handling it right now. Thanks.