[Discover] - Add Breakdown Field Option in Field Stats Popover

ninoslavmiskovic commented 2 months ago

WHY:

In Discover, users frequently need to break down data by specific fields. While this is currently possible through the histogram, there is no way to initiate a breakdown directly from the field stats popover. Adding this capability would streamline workflows, allowing users to apply breakdowns more quickly without navigating away from the field stats view.

WHAT:

Introduce an “Add Breakdown” option within the field stats popover, supporting both KQL and ES|QL:

For KQL Users: The “Add Breakdown” would update the breakdown in the existing visualizations, making it easier to apply a field-based breakdown directly from the stats popover, without altering the main query. Below is an example of the current histogram with a breakdown of the log level.

For ES|QL Users:

For ES|QL, it will be achieved by the same way as in DSL mode with one exception. If STATS is already present in the ES|QL query, then hide the button.

Supported Field Types for Breakdown:

The breakdown functionality should support any field type that can be aggregated in Elasticsearch for DSL mode. This includes:

•   Keyword
•   Numeric
•   IP
•   Boolean
•   Binary

These field types are commonly used for aggregation in Elasticsearch, making them ideal for breakdowns. When a user selects one of these fields in the stats popover, the system should enable the “Add Breakdown” option, allowing for segmentation based on that field.

For ESQL we have a broader usage, the isESQLColumnGroupable can be used to find the groupable fields.

BENEFITS:

Improved Efficiency: Users can quickly break down data by specific fields without needing to manually modify the query. KQL users can update visualizations easily, while ES|QL users can directly adjust queries for more complex aggregations.
Simplified Workflow: This feature reduces the number of steps needed to apply a breakdown, making the process more intuitive. Users won’t need to switch between views, as the breakdown can be applied directly from the field stats popover.
Better Data Exploration:By adding breakdowns within the field stats popover, users can easily explore different segments of their data. KQL users will benefit from quick field-based filtering, while ES|QL users can leverage the full power of advanced aggregations and statistical analysis.

elasticmachine commented 2 months ago

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

davismcphee commented 2 months ago

@stratoula do we need inline stats to support this in ES|QL or is it possible today?

stratoula commented 2 months ago

I need to understand better the ER first to reply.

@ninoslavmiskovic in your screenshot and description you mention this

FROM logs-*-*
| STATS COUNT(*) BY log.level

which means I am changing the query of Discover regardless what I have in the editor atm. Which is weird and not what the users are expecting. This suggestion doesn't need inlinestats but I don't think we should do it.

What it would make sense is that when the users are clicking a field then we update the current query with a stats something like

 STATS var0 = count(geo.dest) BY geo.dest

this makes more sense to me and is not blocked by inlinestats

If we want by clicking this to update the histogram and only (add a breakdown) this needs inlinestats but I don't like it as the main query is independent of the histogram query so the users dont get what is done. Adding a breakdown in the histogram can be accomplished when we have inlinestats from the inline editing flyout. I wouldn't add this functionality to the field list.

ninoslavmiskovic commented 2 months ago

when the users are clicking a field then we update the current query with a stats something like

@stratoula This is what I wanted. Similar to what we do with WHERE. Just add the right syntax to the query.

Question: when we get inline stats - could we both update the query and select the breakdown by in the histogram ?

ninoslavmiskovic commented 2 months ago

In dataview mode we could simply update the histogram. I guess ?!

stratoula commented 2 months ago

Question: when we get inline stats - could we both update the query and select the breakdown by in the histogram ?

No because stats in the initial query displays the aggregation chart and not the histogram

ninoslavmiskovic commented 2 months ago

@stratoula

I have changed the GH issue description to add a STATS and a note which would produce an aggregated chart instead of the histogram.

QQ:

Currently, it is possible to create a histogram in ES|QL with a break-down by:

FROM logs-*-*
| EVAL timestamp=DATE_TRUNC(1 hour, @timestamp)
| STATS count(*) BY log.level, timestamp
| RENAME timestamp as `@timestamp every 1 hour`

This creates this histogram visualization:

And also updates the "Break-down by" in the inline editor.

How would this work with INLINE stats ? What would the query be like and the UX in the inline editor ?

stratoula commented 2 months ago

This creates the histogram but only for the specific field, we want the users to also be able to choose among other fields this is why we want inline stats instead of stats. With that being said the query is the same but instead of stats you use inlinestats. I have an implementation here already https://github.com/elastic/kibana/pull/189630 but is blocked by inlinestats being available only on snapshot releases. And this is a step forward https://github.com/elastic/kibana/pull/190213 and initializes the histogram with the log.level but also allows the users to choose a field from the inline editing

ninoslavmiskovic commented 2 months ago

This creates the histogram but only for the specific field, we want the users to also be able to choose among other fields this is why we want inline stats instead of stats

Super clear ! Makes sense.

Great with the links to the issues.

@stratoula WDYT of the current specification with doing GROUP by with STATS instead ? and then when we are ready we can add the break-down with in-line stats. This would decouple the efforts like you mentioned.

stratoula commented 2 months ago

We are also blocked by this https://github.com/elastic/kibana/issues/191988 so we need to wait for both.

ninoslavmiskovic commented 2 months ago

@stratoula I added this part: If STATS is already present in the ES|QL query, then hide the button.

ninoslavmiskovic commented 2 months ago

The first step on ES|QL mode will be to add the break-down and @stratoula is working on this here: https://github.com/elastic/kibana/issues/186369 which is not longer blocked. After that work is finished, the break-down on field stats pop-over can be added.

elasticmachine commented 2 months ago

Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs)

patpscal commented 1 month ago

Hi! I'm validating the designs (particularly the new icon) with Platform. We should be done soon. https://github.com/elastic/kibana/issues/197464

patpscal commented 1 month ago

Hi @ninoslavmiskovic @gbamparop, here are the design details

Instructions:

New icon for button:
Add the icon to the EUI repo Instructions
Specs on Figma
The tooltip copy needs to be reviewed. I tagged @mdbirnstiehl in case he can tackle it once he's back from PTO

Additional Notes:

The current popover design may benefit from some clarity regarding the functionality of the "plus" button, especially when adding to the table versus adding as a filter. They use the same icon, and it's currently quite confusing.

It could also use some design refinement to ensure every action is easily accessible and affordable. Looking ahead, considering the use of an overflow menu for less common actions could be an interesting approach.

Here are some early explorations cc/ @l-suarez

Figma link

gbamparop commented 1 month ago

Additional Notes:

The current popover design may benefit from some clarity regarding the functionality of the "plus" button, especially when adding to the table versus adding as a filter. They use the same icon, and it's currently quite confusing.

It could also use some design refinement to ensure every action is easily accessible and affordable. Looking ahead, considering the use of an overflow menu for less common actions could be an interesting approach.

Here are some early explorations cc/ @l-suarez

Thanks @patpscal! To clarify, these suggestions are for a follow-up and not blocking the current issue right?

patpscal commented 1 month ago

Yes @gbamparop, at least from my side I would defer them to future improvements

ninoslavmiskovic commented 1 month ago

@l-suarez would love your feedback here.

@MichaelMarcialis I remember there were some hesitations around the filter icons previously, so we did not change them.

Also feel free to provide feedback.

ninoslavmiskovic commented 1 month ago

I will take a look @patpscal 👍

MichaelMarcialis commented 1 month ago

I remember there were some hesitations around the filter icons previously, so we did not change them.

Correct, there were some past concerns about using the filterInclude and filterExclude icons from the icon library. The visual refresh team and I have it in our list of priorities to refresh our entire icon library. These two icons in particular are marked to be redesigned as part of this effort. This task is in progress at the moment and is expected to be completed for the release of the upcoming Borealis EUI theme.

l-suarez commented 3 weeks ago

Thanks @patpscal, great work. As Michael mentions, we will wait until the refresh is done to get the new iconography in place. For now, could we use the new icon that Patri proposes? It represents the histogram action more accurately, rather than the pie chart.

We're adding a new action to the group in the field popover, so currently we display four. Which is fine. If in the future we need to add new ones, I recommend to go for the overflow example that Patri suggests. Main 1 or 2 actions visible and secondary actions under a more menu.

cc / @ninoslavmiskovic

elastic / kibana