zio / zio-metrics-legacy

⛔️ DEPRECATED
https://zio.github.io/zio-metrics
Apache License 2.0
56 stars 39 forks source link

statsd metrics are not sent in batches #122

Open andrewhamon opened 2 years ago

andrewhamon commented 2 years ago

As best I can tell, each metric is sent one by one, not in any batched or buffered way. Does the bufferSize param and the call to groupedWithin serve any useful purpose?

toxicafunk commented 2 years ago

The idea here was to have 2 different listen functions. A default one that uses the udp function which sends the message one by one and another in which you can pass your own function to process the metrics as you best consider. On both cases, groupedWithin(bufferSize, duration) would create batches of metrics either by bufferSize or by time elapsed since the last batch was processed (duration param).

So, groupedWithin creates batches which udp(the default function) sends one-by-one (within the batch) or according to whatever function you pass to listen. Hopefully none of this was broken upon refactoring.

ollyw commented 2 years ago

It sure would be good to have the metrics sent in batches and the UDP buffer being larger than 512. The payload size for UDP that is guaranteed to be delivered is actually 508 bytes, not 512 (https://stackoverflow.com/questions/1098897/what-is-the-largest-safe-udp-packet-size-on-the-internet), but realistically the internet runs with an MTU of 1500 (https://blog.benjojo.co.uk/post/why-is-ethernet-mtu-1500) and internal infrastructure can have much larger MTU.

Using the ZStream to coalesce serialised metrics seems practial, rather than using ZStream accumulate unserialized metrics (some of which will be discarded to to sampling) seems preferable.