search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.35k stars 7.98k forks source link

[Meta] Gather SLO/SLI + ES|QL requirements #182995

Open jasonrhodes opened 1 week ago

jasonrhodes commented 1 week ago

We know that we currently allow users to define SLIs using KQL statements to filter events, to identify good events, and to identify the total number of events.

Screenshot 2024-05-08 at 4 06 24 PM

As we move toward embracing ES|QL more consistently across the product, we should examine how SLI and SLO definitions might change if they were to support ES|QL as the definition language. Note: These requirements should assume that ES transforms already support ES|QL, even though they currently do not. This is true so that we can identify what would be required of us to support ES|QL once transforms are able to support it, in order to help clarify our requirements of transforms when it comes to ES|QL support and to be as ready as possible once they do.

### Tasks
- [ ] Form design for SLI + ES|QL
- [ ] Query design when ES|QL is passed to transform definition
elasticmachine commented 1 week ago

Pinging @elastic/obs-ux-management-team (Team:obs-ux-management)

sophiec20 commented 1 week ago

A better path would be for ES|QL to support materialized views. cc @tylerperk @uric -- please do not assume that ES|QL support in transforms is confirmed.

kdelemme commented 1 week ago

I think we would need to create a new SLI for ES|QL specifically to avoid bloating more the current KQL one. Then it would be just a matter of storing two ES|QL query or at the very least the where clause for each, one representing the good and one the total query.

simianhacker commented 3 days ago

Assuming ES|QL will support "materialized views" and that Transforms will probably not support ES|QL... I imagine we could create a solution that "feels" mostly seamless to the user through the SLO UI (and API).

  1. User choses ES|QL Indicator for the SLI type
  2. Presented with a form to enter their ES|QL query (along with a preview of the results)
  3. We either require the user to generate a result set with @timestamp, goodEvents, totalEvents OR have a form where they can map the output fields to the correct destination.
  4. In the background, we convert the ES|QL query into a materialized view which outputs to a destination index
  5. We then create a SLI transform to read the data from destination index of the materlized view and everything else works the same

There are a lot of unknowns here... We have no idea at this stage if you could take an ES|QL query (as we know it today) and easily convert it to a materialized view OR if there is something that has to happen with the syntax of the query itself. We'd also need to workout who manages the destination index, probably the admin since there could be security considerations beyond our comprehension.

jasonrhodes commented 3 days ago

A better path would be for ES|QL to support materialized views. cc @tylerperk @uric -- please do not assume that ES|QL support in transforms is confirmed.

@sophiec20 thanks, I had a meeting with @dimkots and @prwhelan last week to make sure we are communicating about this situation, and that meeting led to me creating this ticket. They're going to create a similar meta ticket for the ML side, and we're going to collaborate by way of sharing requirements and looking for the best solution from there.

At the end of the day, for SLOs, it might not be important whether transforms directly support ES|QL or if another feature such as "materialized views" provides the bridge for those things to work together, so long as the ES|QL-related user requirements can be met for the SLO product. So we need to be very clear about what our exact SLO user requirements are, where ES|QL is concerned. That's the purpose of this meta ticket.

jasonrhodes commented 2 days ago

or at the very least the where clause for each, one representing the good and one the total query

@kdelemme yeah, this is one of the ideas I think we need to try to explore a bit, because I'm not sure if it's all that easy to "split up" ES|QL phrases into sub-phrases like this, or how that would work/look ...