[Lens] Discover drilldown improvements

[stratoula]

Describe the feature:

At the moment Lens support the Explore data in Discover functionality which works like that:

• The unified search state is being carried in Discover
• The fields (columns) which take place in the visualization are also being carried
• The top values / filters configuration is also being carried but only if Other switch is not selected

The latter creates a lot of confusion to the users:

• It is difficult to understand why de-selecting the Other switch creates this ugly kql query
• The query can be big so it is difficult to be edited
• If the terms are quite a lot, the url in Discover will get super long, causing the browser to get to the limit and Discover displays an error.

For example this chart

leads to this Discover page

I feel that we should re-consider this feature.

• Do we really need to navigate users to Discover? Maybe this is a very good opportunity of implementing a way to visualize a table of data in Lens. One idea is to open a flyout which shows the underlying documents without rendering the KQL query (or filter) which was used to filter this data.
• The Explore data in Discover should remain there but the functionality should change to (it should also be renamed in something else such as go to discover):
  • The unified search state is being carried in Discover
  • The fields (columns) which take place in the visualization are also being carried

[elasticmachine]

Pinging @elastic/kibana-visualizations (Team:Visualizations)

[drewdaemon]

My 2 🪙

The point of this feature is to show the documents that are powering the visualization. Not filtering down to this particular set of documents will cause the user to see documents that aren't used in the visualization. Are we comfortable with that?

Possibly it is a naming problem — perhaps the ambiguity of "Explore data in Discover" which we used instead of "View underlying data" has opened the door to alternative interpretations of this functionality by customers, etc.

That said, I think the main objections here come from making details that are quiet in Lens loud and scary(?) in Discover via the big KQL query.

I like the idea of the flyout — it keeps the details hidden while still showing the right documents.

Alternatively, we could transfer the state to Discover as a filter pill instead of a big KQL query. That would give us a chance to assign a label to the filter and wouldn't interfere with the use of the search bar.

[dej611]

Agree with @drewdaemon here.

I can add also something about iterating over the Explore concepts: I think that both "carry over" all the charts filter into Discover AND "carry over" only unified search context into Discover are valuable and perhaps both of them can be pursued with a redesign of the feature. In a way we could offer multiple options to "Explore in discover" which let the user decide what to carry and what not, same as it happens now with the sharing functionality that lets user decide whether to carry over the time range or not. Options that can be offered, in my idea:

• current filtering context (search bar + filters)
• current time range
• current filters defined within chart configuration (operation advanced filters, formula's Lucene/KQL filters, Filters, Terms without Other, etc...)

[drewdaemon]

Actually, another nice thing about the flyout idea is that we could support the multi-layer case. Right now, we just disable "Explore data in Discover" if there is more than one layer in the vis.

[stratoula]

Even if we replace it to a filter is still very difficult to handle it for many terms. Also doesn't solve the many terms url problem. So tbh I really vote for another UI because I agree that it is a nice functionality, it is just our implementation that is a bit problematic.

[drewdaemon]

Another thing the current implementation does is add disabled filters to Discover when some dimensions are a subset of another dimension.

For example, I can have Count of Apples, Count of Oranges, and Total count of fruit as three dimensions in a line chart. When I click "Explore data in Discover," I get two disabled filters "fruit: Apple" and "fruit: Orange." The idea was that the visualization uses all the documents because of the "Total count of fruit" dimension, but the user might want to see only the documents powering the "Count of Apples" line. If so, they just turn on one of the pre-built filters.

[drewdaemon]

Also wondering if this situation got worse as a side effect of https://github.com/elastic/kibana/issues/162456.

[kertal]

Do we really need to navigate users to Discover? Maybe this is a very good opportunity of implementing a way to visualize a table of data in Lens. One idea is to open a flyout which shows the underlying documents without rendering the KQL query (or filter) which was used to filter this data.

A flyout showing underlying documents would IMO be the better experience. Initially we intended to migrate also data fetching to the UnifiedDataTable

https://github.com/elastic/kibana/tree/main/packages/kbn-unified-data-table

But due to shift of priorities and time constraints, we couldn't. I wonder if the upcoming improvements of the embeddable API could allow us to create saved searches / discover views on the fly which could be a useful functionality in more areas of Discover. Also Dashboard, having such a flyout to show the underlying documents of a visualization could be useful.