[Observability AI Assistant] Add tests for serverless projects

[miltonhultgren]

Summary

With most of our features being available in serverless Observability projects we should have some test automation in place to make sure it doesn't break without us noticing.

Adding tests for serverless is the same as adding tests for stateful but in a different folder xpack/test_serverless/functional/test_suites/observability but these test suites cannot rely on any test tools available in stateful tests.

So we need to decide on either duplicating all of our setup and helper code or extract that code into packages and refactor the existing stateful tests to use those packages instead before adding serverless tests.

Notes

While testing ELSER (TINY_ELSER) in a Serverless project, we need to call await ml.api.assureMlStatsIndexExists(); before importing the model, not after as the other (non-serverless) ML tests show.

[elasticmachine]

Pinging @elastic/obs-knowledge-team (Team:obs-knowledge)

[neptunian]

Tests were duplicated for API integration in https://github.com/elastic/kibana/pull/192219. Many were skipped and below are issues still needed to address:

• [ ] https://github.com/elastic/kibana/issues/192757
• [ ] https://github.com/elastic/kibana/issues/192886
• [ ] https://github.com/elastic/kibana/issues/192751
• [ ] https://github.com/elastic/kibana/issues/192701
• [ ] https://github.com/elastic/kibana/issues/192497
• [ ] https://github.com/elastic/kibana/issues/192711
• [ ] https://github.com/elastic/kibana/issues/192718
• [ ] serverless functional tests
• [ ] inquire with ml-ui-team to have the ability to delete system indices which we do after uninstalling tiny elser with .ml indices

[neptunian]

QAF issue to add custom config to the MKI pipeline https://github.com/elastic/qaf-tests/issues/126