search

Shopify / statsd-instrument

A StatsD client for Ruby apps. Provides metaprogramming methods to inject StatsD instrumentation into your code.
http://shopify.github.io/statsd-instrument
MIT License
570 stars 94 forks source link

Add statistics to the BatchedUDPSink #367

Closed ahayworth closed 3 months ago

ahayworth commented 3 months ago

This commit tracks a few simple statistics around the performance of the BatchedUDPSink:

We add a new option STATSD_BATCH_STATISTICS_INTERVAL (default: 0), which when set to any non-zero value, will track and flush statistics at that same interval.

To do this, we need to use a mutex for synchronization. Local benchmarks indicate that the performance impact is minimal, if not non-existent:

zsh ❯ ./send-metrics-to-dev-null-log
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
Warming up --------------------------------------
StatsD metrics to /dev/null log (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166)
                        11.805k i/100ms
Calculating -------------------------------------
StatsD metrics to /dev/null log (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166)
                        129.435k (± 1.9%) i/s -    649.275k in   5.018026s

Comparison:
StatsD metrics to /dev/null log (branch: ahayworth/batched-udp-stats-wow-such-meta, sha: cb6297d):   130488.2 i/s
StatsD metrics to /dev/null log (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166):   129434.7 i/s - same-ish: difference falls within error
zsh ❯ ./send-metrics-to-local-udp-receiver
===== UDP sync =====
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
Warming up --------------------------------------
UDP sync (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166)
                         2.173k i/100ms
Calculating -------------------------------------
UDP sync (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166)
                         20.776k (± 7.6%) i/s -    104.304k in   5.058975s

Comparison:
UDP sync (branch: ahayworth/batched-udp-stats-wow-such-meta, sha: cb6297d):    21049.0 i/s
UDP sync (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166):    20776.0 i/s - same-ish: difference falls within error

================
===== UDP batched =====
ruby 3.3.0 (2023-12-25 revision 5124f9ac75) [arm64-darwin23]
Warming up --------------------------------------
UDP batched (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166)
                         8.699k i/100ms
Calculating -------------------------------------
UDP batched (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166)
                         86.768k (± 1.6%) i/s -    434.950k in   5.014276s

Comparison:
UDP batched (branch: ahayworth/batched-udp-stats-wow-such-meta, sha: cb6297d):    87104.6 i/s
UDP batched (branch: main, sha: https://github.com/Shopify/statsd-instrument/commit/3c944c95d103e84cd04671c27dc57e965ad21166):    86767.6 i/s - same-ish: difference falls within error

Of course, we are doing more work when the option is enabled, so technically there must be some impact to it. However, at least in local micro-benchmarks, it certainly seems acceptable.

ahayworth commented 3 months ago

We ran this for a bit in production, and everything looks correct and nothing imploded. I'm going to merge it; we may want to add additional stats in the future though.