Metrics improvments

[swirlds-automation]

Our current metrics system has a number of things that need to improve. These improvements can be split up into a number of categories:

Metrics instances

• 5375

• 9801

Metrics framework

All issues related to the metrics framework. Some features might be missing, some metrics types might be needed.

• Cycle metrics need some improvement, like adding the number of cycles per second. (I have a lock-free per second metric in a branch, needs some more work)
• make metric descriptions mandatory

Grafana

• We should have the option to overlay multiple metrics on the same graph
• We should have logarithmic scaling
• There should be functions for smoothing out some metrics, like calculating a running average, or merging sampling periods

Metrics monitoring

• We should have tools that monitor lots of metrics on all our environments (but mainnet especially) and take appropriate action depending on the environment. The same criteria should be used for validating JRS and local test metrics

[swirlds-automation]

It would also be good if our graphing of the metrics took advantage of the fact that our metrics already calculate running average, max, min, and standard deviation.

For example, if we graph the max of all the 3-second samples of the running average, that isn’t as good as graphing the max of all the maxes of the data that was recorded within the 3 seconds. So the graphing should be the max of the maxes we are are calculating in the metric. Rather than just the max of the averages.

_{author:lbaird, createdAt:2023-01-17T14:58:42Z, updatedAt=2023-01-17T14:58:42Z}

[swirlds-automation]

New metrics should not be added manually to grafana

Instead of using Grafana for your use case (have an automatic graph view of all metrics) we should use Prometheus for that. I written down an idea how that can be achieved in https://github.com/swirlds/swirlds-platform/issues/6550 _{author:hendrikebbers, createdAt:2023-01-18T08:17:11Z, updatedAt=2023-01-18T08:17:11Z}

[swirlds-automation]

It would also be good if our graphing of the metrics took advantage of the fact that our metrics already calculate running average, max, min, and standard deviation. For example, if we graph the max of all the 3-second samples of the running average, that isn’t as good as graphing the max of all the maxes of the data that was recorded within the 3 seconds. So the graphing should be the max of the maxes we are are calculating in the metric. Rather than just the max of the averages.

There are some challenges still with the metrics you wrote, we can discuss this on Friday or even before that if you have time. Or, if you prefer, I can write down what the challenges and tradeoffs are.

_{author:lpetrovic05, createdAt:2023-01-18T10:10:58Z, updatedAt=2023-01-18T10:10:58Z}

[swirlds-automation]

_{migrated from: url=https://github.com/swirlds/swirlds-platform/issues/6549 author:lpetrovic05, #:6549, createdAt:2023-01-17T11:12:27Z, updatedAt=2023-02-17T22:55:29Z labels=Improvement,Migration:Hashgraph,Migration:Base}