search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.8k stars 8.19k forks source link

[Lens] be smarter when building open-in-discover query based on formulas #167352

Open drewdaemon opened 1 year ago

drewdaemon commented 1 year ago

Open in Discover doesn’t currently assign semantic meaning to arithmetic operations within a formula. Right now, it just sends along all the formula queries it detects separated by OR statements.

However, this does not always reflect the documents that actually make up the formula result. Take the following percentage formula

1 - count(kql='fruit.color : red') / count(kql='fruit.color : *))

Say the count of red fruit is 3 and the total count is 10.

The resulting number is 0.7 (70%).

Open in Discover will result in the following query

( ( fruit.color : red ) OR ( fruit.color: * ) )

Two problems with this

  1. the 70% is actually composed of the documents where fruit.color is NOT red
  2. the inclusion of the fruit.color: * filter makes the fruit.color : red filter meaningless because the latter is a subset of the former
elasticmachine commented 1 year ago

Pinging @elastic/kibana-visualizations (Team:Visualizations)

stratoula commented 1 year ago

I really think that we should not try to create kql query from the formula when navigating to Discover. Otherwise we will have cases like this. Formulas already support the Filter by and I think that is the only one that should be transferred.

image
dej611 commented 1 year ago

the inclusion of the fruit.color: * filter makes the fruit.color : red filter meaningless because the latter is a subset of the former

I think this has been already discussed in the past and decided that was ok. Maybe we could improve it somehow but it can be a rabbit hole.

Formulas already support the Filter by and I think that is the only one that should be transferred.

That was added later on, therefore there's the chance some formulas contain all filters within the formula text. Also, this won't change the original problem of "showing something else" in Discover, it just move the targets from "sometimes show the opposite set" to "more often show a way bigger set than in the visualization".

As per cases like that, I do not think they are the majority. As in the case above the same expression can be translated with the not operator: count(kql='not fruit.color : red') / count(kql='fruit.color : *)) and have the correct filter in Discover.

I see the problem here, but I suspect that any "shortcut" we might take will have no matter what a drawback, so I would propose to start making the user aware of what the transition would do. It can be a message in a modal, a help tooltip or just a tooltip on the Explore button, or a combination of these, who explains that all filters contained in the visualization would be transitioned in Discover.

stratoula commented 1 year ago

@dej611 I have it as a discuss point next Monday. I am not persuaded that we should transfer everything. Especially the formulas filters do not make sense to me tbh. But let's discuss. I want to hear the reason we decided to do that in the past.

stratoula commented 1 year ago

Discussed offline with the team. What we could do is for cases like the above is to notify the users about which will be the KQL query we are going to create and then they can tweak it. It needs design if we decide to do this at some point in the future.