[ML] Enhance log rate analysis API for use with AI Assistant

[grabowskit]

Log rate analysis is a Kibana API that uses several steps to get its relevant fields. This is too slow for the AI Assistant to be able to call and needs an updated API that can be called ideally in a single step depending on feasibility.

@dgieselaar

### Dependencies
- [ ] https://github.com/elastic/kibana/issues/178613
- [ ] https://github.com/elastic/ml-team/issues/1182#issuecomment-2107506340

[elasticmachine]

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[walterra]

I created an additional issue that will cover this use case here: https://github.com/elastic/kibana/issues/178613

Let me know if you have any suggestions about the API request body and what you'd like to pass on when calling this via AI Assistant.

[walterra]

Can you elaborate a bit on the requirements regarding speed related to AI Assistant? Are there some kind of thresholds, what does the AI Assistant consider too slow?

[dgieselaar]

@walterra I think this API should not take more than 2-3s to return. Generally I think we want more control here over the actual analysis that is being performed - e.g. there are some steps where we can take advantage of the fact that we know the schema to speed things up/get back more useful results.

[mwtyang]

cc @arisonl

[emma-raffenne]

@walterra and @sorenlouv Please can you sync on the progress of this and the requirements from our side for the AI Assistant? Thanks

[walterra]

We also have an issue in the ml-team repo to track work from our side which I updated just yesterday: https://github.com/elastic/ml-team/issues/1182#issuecomment-2107506340

So far I didn't hit any limitations regarding the AI Assistant, the recent updates with starter prompts are great and I was able to now use setScreenContext properly to pass on custom data from Kibana pages. More details can be found in issue I linked.

[sorenlouv]

@walterra I'd like to try out the log rate analysis in a rest API that will be consumed by the AI Assistant. If I pull your branch, how can I invoke the log rate analysis function that your proposed here?

[walterra]

Summarizing our ZOOM here:

• The current PoC so far focused on a registered function for the AI Assistant.
• The use case Søren is after is to make log rate analysis available for the "contextual insight" variant of the AI assistant.

In that case it doesn't need to be a registered function, it just needs to be a function that can be used on Kibana's server side as part of the API call that gathers the data for the initial prompt. I'll try to expose such a function from the @kbn/aiops-log-rate-analysis package to get us started.