[Security Assistant] Automatically Install Knowledge Base

Summary

This PR is Phase 1 of the Knowledge Base work for 8.15, which includes automatically setting up the Knowledge base (this PR), introducing new generic KB tools for recall/retrieval, a CRUD API for for managing Knowledge Base Entries, and a basic UI for Knowledge Base Entry management (all captured in this issue). Once complete, this will also provide the opportunity to remove the !isEnabledKnowledgeBase code paths, directing all interactions through our LangChain Agent pipeline.

This PR sets the ground work for all of the above by moving ELSER setup and Knowledge Base data management to use the new AssistantDataClient architecture used for Conversations, AnonymizationFields and Prompts.

This feature is currently behind the assistantKnowledgeBaseByDefault experimental feature flag, which can be enabled by adding the following to your kibana.dev.yml:

xpack.securitySolution.enableExperimental:
  - 'assistantKnowledgeBaseByDefault'

Once enabled, an Install Knowledge Base button will be shown when starting a new conversation. Note: UX is still under development.

Useful Dev Tools Queries

The new assistantKnowledgeBaseByDefault flows are quite resilient, and so everything should function as expected even if one piece of the puzzle is missing or incomplete. Here are some dev tool queries to check and delete individual resources, which is nice for testing. For instance, you can nuke the ingest pipeline, or ELSER, and the Install KB button will appear and function as intended.

[!NOTE] Since the existing API's were used, with forked logic for the assistantKnowledgeBaseByDefault FF, the existing KB Settings UI still functions as expected, and can be used for deleting and re-initializing the KB. This functionality will most likely go away with updates to the KB UI, but is nice for testing in the interim.

Useful Dev Tools Queries

``` ts // New KB GET /_ingest/pipeline/.kibana-elastic-ai-assistant-ingest-pipeline-knowledge-base GET /_index_template/.kibana-elastic-ai-assistant-index-template-knowledge-base GET /_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default/ GET .kibana-elastic-ai-assistant-knowledge-base-default/_count GET .kibana-elastic-ai-assistant-knowledge-base-default/_mapping GET .kibana-elastic-ai-assistant-knowledge-base-default/_search { "size": 1000 } // MSearch for if ES|QLKB docs exists GET .kibana-elastic-ai-assistant-knowledge-base-default/_msearch {} {"query":{"bool":{"must_not":[{"term":{"metadata.kbResource":"esql"}},{"term":{"metadata.required":true}}],"must":[{"text_expansion":{"vector.tokens":{"model_id":".elser_model_2","model_text":"You can chain processing commands, separated by a pipe character: `|`."}}}]}},"size":10} {} {"query":{"bool":{"must":[{"term":{"metadata.kbResource":"esql"}},{"term":{"metadata.required":true}}]}},"size":10000} // Other DataClient Assets GET .kibana-elastic-ai-assistant-anonymization-fields-default/_search { "size": 1000 } GET .kibana-elastic-ai-assistant-conversations-default/_search { "size": 1000 } GET .kibana-elastic-ai-assistant-prompts-default/_search { "size": 1000 } GET /_ingest/pipeline/.kibana-elastic-ai-assistant-ingest-pipeline-knowledge-base // Delete them all! // Data Streams DELETE /_data_stream/.kibana-elastic-ai-assistant-anonymization-fields-default DELETE /_data_stream/.kibana-elastic-ai-assistant-conversations-default DELETE /_data_stream/.kibana-elastic-ai-assistant-knowledge-base-default DELETE /_data_stream/.kibana-elastic-ai-assistant-prompts-default // Index Templates DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-anonymization-fields DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-conversations DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-knowledge-base DELETE /_index_template/.kibana-elastic-ai-assistant-index-template-prompts // Pipelines DELETE /_ingest/pipeline/.kibana-elastic-ai-assistant-ingest-pipeline-knowledge-base ```

New Features:

[X] Plumbed through new assistantKnowledgeBaseByDefault experimental feature flag and exposed through assistantCapabilities API
[X] Cleaned up assistantFeatures made available in AssistantProvider and tests (no need to individually plumb new features)
[X] Introduced new AIAssistantDataClient for creating Data Streams, Component Templates, and Ingest Pipeline
[X] Use AIAssistantDataClient to automatically install ELSER via installElasticModel() API, then deploy via TrainedModelsAPI
[X] Plumb through addKnowledgeBaseDocuments() for creating KB entries from LangChain Documents within AIAssistantDataClient
[X] Update ElasticsearchStore to take a kbDataClient for use in document adding/retrieval

Changes not behind FF:

[X] Updated getELSER() helper function to be called by internalUser to prevent ml privilege requirements to setup Knowledge Base once a privileged user has enabled ELSER
[X] Updated get/post/delete knowledge base routes to create esStore as internalUser to enable all assistant users the ability to enable the KB once ELSER has been installed (currently they 503 if currentUser doesn't have read_ingest and manage/read for the .kibana-elastic-ai-assistant-kb index)
[X] Updated get/post/delete knowledge base routes with assistant API access controls: tags: ['access:elasticAssistant'],
[X] Relaxed id validation from UUID to UUID or NonEmptyString to support ES generated id's, see: ddf93a8

Checklist

Delete any items that are not applicable to this PR.

[X] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
[ ] Documentation was added for features that require explanation or tutorials
- TBD once FF is removed
[ ] Unit or functional tests were updated or added to match the most common scenarios
- Existing tests updated with new functionality, new tests to be fleshed out as feature stabilizes before removing FF

:green_heart: Build Succeeded

Buildkite Build
Commit: 239f8342609fade647374165a3cc250ad7bf821a

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	5492	5497	+5

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/elastic-assistant`	147	138	-9
`@kbn/elastic-assistant-common`	217	275	+58
total			+49

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	15.1MB	15.1MB	+2.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`securitySolution`	83.5KB	83.5KB	+35.0B

Unknown metric groups

#### API count | id | [before](https://github.com/elastic/kibana/commit/135961b720657960f7648887246383833645d7e3) | [after](https://github.com/elastic/kibana/commit/239f8342609fade647374165a3cc250ad7bf821a) | diff | | --- | --- | --- | --- | | `@kbn/elastic-assistant` | 174 | 165 | -9 | | `@kbn/elastic-assistant-common` | 232 | 292 | +60 | | total | | | +51 | #### ESLint disabled line counts | id | [before](https://github.com/elastic/kibana/commit/135961b720657960f7648887246383833645d7e3) | [after](https://github.com/elastic/kibana/commit/239f8342609fade647374165a3cc250ad7bf821a) | diff | | --- | --- | --- | --- | | `elasticAssistant` | 38 | 39 | +1 | #### Total ESLint disabled count | id | [before](https://github.com/elastic/kibana/commit/135961b720657960f7648887246383833645d7e3) | [after](https://github.com/elastic/kibana/commit/239f8342609fade647374165a3cc250ad7bf821a) | diff | | --- | --- | --- | --- | | `elasticAssistant` | 38 | 39 | +1 |

History

:green_heart: Build #210848 succeeded 17f633ea851560912ac030b3c3815226160ffa3a
:green_heart: Build #210261 succeeded bef08b7a52257daca03d0042a454012ec94ad03f
:broken_heart: Build #210137 failed 146aafb49081b1529064b4b9fa36f25999969dee
:broken_heart: Build #210032 failed 684488a952bdd4dce77331609f5d1b62a2ba4581
:broken_heart: Build #210028 failed 2b10c9ca2973136676f3ee9622c461513ec31357
:broken_heart: Build #210027 failed be327c114f65e0f7f9c0c3ebef13c23ed7d10326

To update your PR or re-run it, just comment with: @elasticmachine merge upstream

cc @spong

elastic / kibana