Kolla: Monitoring High Availability

[linwalth]

With the current state of Kolla-Ansible it is not possible to run multiple Monitoring Servers in HA which creates problems with Prometheus (to a lesser extent) as well as Gnocchi and Ceilometer (probably among others).

The problems are as following:

• Multiple Prometheus Servers scraping the same endpoints generates undue load on the exporters, e.g. the Openstack-Exporter will scrape thrice - thus generating thrice the load on Keystone.
• Multiple Prometheus Servers scraping will inevitably generate slightly differing datasets, each on the servers they reside on.
• Multiple Gnocchi Backends on the Monitoring Servers will create wildly inconsistent datasets due to loadbalancing between the Monitors
• This probably also pertains to Ceilometer utilizing Prometheus as storage Backend due to the inconsistent datasets (save for if the Pushgateway sends the data to each Prometheus instance, which I do not know)

It would be great if Kolla Monitoring was able to leverage something like Thanos to store the data in RGW as well as make it possible to operate Monitoring Server HA.

[berendt]

The SCS observablity stack can be used for this in the future. Kolle monitoring will not be improved.