[Monitoring] Adapt UI code to read from metricbeat-* indices

[chrisronline]

Relates to https://github.com/elastic/beats/pull/19747

In an effort to ensure a smooth migration path to the upcoming Agent, we want to adapt all queries in the UI to handle data living in .monitoring-* indices, or metricbeat-* indices. The Beats team is helping us in adapting each stack module to ship directly into metricbeat-* and we need to handle the UI changes to ensure that the UI will work properly.

Work items

• [x] Update mappings to ensure we only map the fields used by the Stack Monitoring UI -> https://github.com/elastic/kibana/pull/76008
• [x] Adapt UI code to read ES node_stats from metricbeat-* -> https://github.com/elastic/kibana/pull/76015
• [ ] Adapt UI code to read ES index_stats from metricbeat-* -> https://github.com/elastic/kibana/pull/90838
• [ ] Adapt UI code to read ES indices_stats from metricbeat-* -> https://github.com/elastic/kibana/pull/90838
• [ ] Adapt UI code to read ES job_stats from metricbeat-* -> https://github.com/elastic/kibana/pull/90838
• [ ] Adapt UI code to read ES shard from metricbeat-* -> https://github.com/elastic/kibana/pull/90838
• [ ] Adapt UI code to read ES ccr_stats from metricbeat-* -> https://github.com/elastic/kibana/pull/90838
• [ ] Adapt UI code to read Kibana kibana_stats from metricbeat-*
• [ ] Adapt UI code to read Logstash logstash_stats from metricbeat-*
• [ ] Adapt UI code to read Logstash logstash_settings from metricbeat-*
• [ ] Adapt UI code to read Beats beats_stats from metricbeat-*
• [ ] Adapt UI code to read Beats beats_state from metricbeat-*
• [ ] Ensure telemetry code reads from metricbeat-*
• [ ] Ensure security concerns are handled https://github.com/elastic/kibana/issues/90660

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[chrisronline]

I have some updates here.

I have written code that will generate the minimal set of mappings necessary for each stack product. I checked my work by modifying all existing es_archive mapping files to use these new mappings, then ensuring all tests pass.

Elasticsearch: https://gist.github.com/chrisronline/a002d91b51f0fdb1beb1aa2af31718b3 Kibana: https://gist.github.com/chrisronline/abcb27a2cca09fd542ef7b042984f005 Beats: https://gist.github.com/chrisronline/abbbbaf723f21c53a2aaaacc9a44000c Logstash: https://gist.github.com/chrisronline/23922534b97b3a001d227d3e772cdb16

Using the above, we should be able to add the minimal mappings to metricbeat-*.

[chrisronline]

I've updated the above mappings to also account for existing stack monitoring cluster alerts and telemetry code too. There were a few additions as a result