Encapsulate access to Inference APIs

[legrego]

kibana_system has the ability to perform inferences. This is a feature used by Kibana's AI assistants.

The ability to perform inferences requires the monitor_inference cluster privilege. From an administrative standpoint, we would prefer that a role only require access to the AI Assistant privilege, rather than require access to both the AI Assistant privilege, and the monitor_inference cluster privilege. The inference APIs are an implementation detail of the AI Assistant feature.

I propose adding an Inference Service, which can:

• Perform authorization checks, to ensure the user has privileges which permit access to one or more AI Assistant features.
• Perform audit logging, to capture the user intent.
• Perform the inference API call using kibana_system credentials on behalf of the end user.

This pattern is very similar to what we've done with Saved Objects, and fits well within our existing authorization model.

We will likely need changes to the features plugin as well, to allow features to register the fact they permit access to inference APIs.

Opening this up for discussion before we commit to an approach.

cc @YulNaumenko @jonathan-buttner @spong @lukeelmers @elastic/kibana-security

---

Note there are also requirements to allow for the automated configuration of inferences, which would require credentials with the manage_inference cluster privilege. This is out of scope of the current discussion.

[spong]

This general approach sounds good to me -- just one question around if we're planning to bundle the 'setup ELSER inference endpoint' responsibility to this inference service as well, or if that will live elsewhere? If so, the manage_inference privilege will be required for creating the new inference endpoint. I suppose manage_inference is needed for the inference connector efforts as well (to create new endpoints), but will let @YulNaumenko speak to that.

As an FYI, we also have this new inference plugin proposal in-flight for abstracting inference tasks (https://github.com/elastic/kibana/pull/188280). Not saying there's overlap or that they should be combined or anything like that, just mentioning for visibility around all things inference over on the Kibana server side of things.

[legrego]

just one question around if we're planning to bundle the 'setup ELSER inference endpoint' responsibility to this inference service as well, or if that will live elsewhere

That is TBD. We still haven't agreed to grant manage_inference to the kibana_system user, so the details of how we accomplish these goals need to be worked out yet.

As an FYI, we also have this new inference plugin proposal in-flight for abstracting inference tasks (https://github.com/elastic/kibana/pull/188280). Not saying there's overlap or that they should be combined or anything like that, just mentioning for visibility around all things inference over on the Kibana server side of things.

Thanks for sharing, that is great to know!