Add automated tests with dashboards

[jsoriano]

Currently the only tests we have with dashboards check that they can be installed during assets tests. Having functional tests would allow us to check that the dashboards actually work, and to have additional checks on the data collected.

We need to find a way to check that the data collected is relevant for dashboards, and that dashboards have enough data to render correctly.

Some ideas to explore:

• Look for programatic ways to check if the dashboard has any panel with errors.
• The inspect view in Kibana dashboards is able to show the queries executed by the panel, if there is some API to access this information we could run the queries during system tests and check that the collected data is valid and relevant for dashboards.
• Add a dashboards test runner (elastic-package test dashboards) that uses data generation like the one provided by elastic-package benchmark stream to test the dashboards.
• Deploy some MITM proxy between Kibana and Elasticsearch and capture queries done when opening dashboards.
• Use webdriver or similar to do headless browser testing.
• Screenshot-based tests could help to validate that the actual rendered dashboard makes sense, but we would need to be able to control all the data represented in a deterministic way.

The implementation could be incremental, adding features in different areas of testing.

cc @lalit-satapathy

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jlind23]

@rayafratkina is this something that your time already gave some thoughts on? Automated tests for dashboards.

[drewdaemon]

Linking to a PoC which adds an API that extracts the Elasticsearch queries from dashboards: https://github.com/elastic/kibana/pull/173416

If this were to be shipped to production in Kibana, the test runner could extract the Elasticsearch queries from integration dashboards and use them to run assertions.

Issue to track: https://github.com/elastic/kibana/issues/175487