[Connectors] Generic Kibana connector for Inference API

[YulNaumenko]

Integrating the new (chat) completions takes significant effort for every new provider. The goal of the current issue is to build the new generic Kibana connector to inference API, instead of existing set of integrations (GenAI connector, Gemini, Bedrock). Inference API will keep integrations at one place. This will simplify the process of adding the new integration, ideally just adding a new provider schema and creating generic inference API connector instance with the set up of the credentials. A user will select a provider supporting the completion task type and Kibana would render the corresponding input form, which would ask for the necessary authentication information (API key etc.) The plan is to use sub-actions as a new inference task type, like chat complete.

Current work could be separated by the smaller efforts/scopes:

• [ ] simplify privileges management for inference API https://github.com/elastic/elasticsearch/pull/108386. The assumption that connector will call inference API on behalf of the Kibana system user.
• [ ] add API to return the list of the existing providers/integrations (in ES or Kibana)
• [ ] add API to return available the schema for the selected provider
• [ ] build Kibana stack-connector which will have a dynamic configuration (Config/Secrets) based on the selected provider.
• [ ] migrate assistants to use inference generic connector instead of the existing connectors
• [ ] deprecate existing connectors

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[kobelb]

/cc @cnasikas

[serenachou]

btw, we're going to need to check into what is happening with the integration "new AI connector" workflow and how it should be using this inference endpoint connector

might be a reason to deprecate the other connectors or accelerate handling migration to the new one