Given the capabilities of InfluxDb 2.7+, investigate wether Prometheus is still required

[pruneau628]

This is a follow-up issue to 61.

61 was raised, because using a metric as a tag in prometheus (or even influxdb) is a anti-pattern.

However, it questions the pertinence of using prometheus, because prometheus is a numeric time series data store, which cannot leverage the textual data provided by indy-node-monitor. I also appeared that influxdb 2.7+ (instead of 1.8) seemed to fill the alerting void left if prometheus was removed from the solution

So we are investigating fully or partly replacing elements of the current (as of 2023-04-11) stack by the latest OSS version of influxdb (currently 2.7.0)

For memory, here is the current architecture:

flowchart TD
    Grafana[/Grafana\n* data display\n* dashboards\] -- queries --> Prometheus
    Grafana -- queries --> InfluxDB
    Prometheus{{Prometheus& \nAlertManager\n* numeric time series\n* alerting}} -- queries --> Telegraf
    Telegraf[[Telegraf\n\n* periodic data gathering\n* data dispatch]] -- stores --> InfluxDB{{InfluxDB\n* numeric time series\n* text-based time series}}
    Telegraf -- queries --> 
    Indy-Node-Monitor[Indy-Node-Monitor\n* status/metrics\ngathering interface]  --> IndyClusters[\Indy Clusters/]
    Indy-Node-Monitor --> IndyClusters
    Indy-Node-Monitor --> IndyClusters

[GuillaumeBourque-QC]

Since Grafana is part of the solution, why not use Grafana's alerting capability to abandon prometheus? And for those who want to do the alerting part at the data level, the new version of influxDB seems to allow alerts ref https://docs.influxdata.com/influxdb/v2/monitor-alert/

I'm not sure what problem we're trying to solve.