search

zio / zio-metrics-connectors

Monitoring, Metrics and Diagnostics for ZIO
https://zio.dev/zio-metrics-connectors
Apache License 2.0
30 stars 24 forks source link

Seperate histogram send frequency #30

Closed ollyw closed 1 year ago

ollyw commented 1 year ago

Typically services want to send metrics at 1, 5, 10, 30 second intervals to minimise storage space. However with distribution metrics, each metric needs sending as they are not sampled. In a high throughput service it is undesirable to store 1000's of metrics waiting when using a low metrics frequency. It consumes memory and makes the service very "bursty", where it sends thousands of metrics with no gap, then pauses for n seconds.

An alternative is to increase the metrics frequency to something like 100ms, which suits the distribution sending, but this causes all the metrics to be queried and sent. There is some suppression of sending metrics that have not changed, but many metrics, such as zio_fiber_fork_locations get updated continuously, but there is little value in having the data at sub-second resolution.

This PR is one approach to decoupling the histogram sending frequency whilst maintaining backward compatibility. I welcome feedback on the approach and welcome suggestions of a better way. @petoalbert, I would value your opinion too!

There are also some small performance optimizations + document updates

petoalbert commented 1 year ago

I like this idea. Originally I was thinking about having a different interval for histogram updates, but I wasn't sure. But you have a valid point about the service being "bursty", so I am convinced this is the right direction.

atooni commented 1 year ago

I also like the idea and am happy to merge once CI is happy as well.

ollyw commented 1 year ago

Ah, that is great. @atooni both this branch and series 2.x branch CI passes. Can we merge?

atooni commented 1 year ago

thx again - merged