Memory leak when using BatchedUDPSink

Shopify / statsd-instrument

A StatsD client for Ruby apps. Provides metaprogramming methods to inject StatsD instrumentation into your code.

http://shopify.github.io/statsd-instrument

MIT License

570 stars 94 forks source link

Memory leak when using BatchedUDPSink #297

Closed vahe closed 2 years ago

vahe commented 2 years ago

We noticed a memory leak on one of our Sidekiq deployments. With rbtrace and diffing heap dumps the indication was it was from statsd-instrument gem.

Retained STRING 755984 objects of size 221275072/228127604 (in bytes) at: /artifacts/bundle/ruby/2.7.0/gems/statsd-instrument-3.1.2/lib/statsd/instrument/datagram_builder.rb:86

After setting STATSD_FLUSH_INTERVAL to 0, the memory leak disappeared (to use the UDPSink instead of BatchedUDPSink)

Memory graphs before:

After:

casperisfine commented 2 years ago

Sorry, somehow I wasn't watching the repo and didn't notice this.

A couple thoughts:

The default 1 second interval is likely too much. We should probably reduce it to 0.1 or something.
We need to handle and upper size to that array. A sleep loop isn't good enough, we need to wake up the background thread if the buffer is past a certain size. I think we could do this with a Monitor, I need to explore that.