search

honeycombio / libhoney-rb

Ruby library for sending data to Honeycomb
Apache License 2.0
11 stars 30 forks source link

Possible memory leak #125

Closed mrsimo closed 1 year ago

mrsimo commented 2 years ago

We recently upgraded honeycomb-beeline from 2.1.1 to 2.10.0, which upgraded libhoney from 1.14.4 to 2.1.0 in one of our rails apps. This change introduced a very noticeable memory leak. After some troubleshooting, pinning the versions of honeycomb-beeline to 2.1.1 and libhoney to 1.18.0 fixed the issue (these are versions used by another of our apps that didn't seem to have a problem).

This is the memory profile of this app's pods during deploys without the memory leak problem:

memory-good

This is as soon as we upgraded:

memory-bad

I took memory heap dumps to troubleshoot the problem and find the culprit, and this is what heapy showed as the top memory usage between two different points in time:

Allocated OBJECT 109 objects of size 7848/2336608 (in bytes) at: /bundle/2021-12-13/ruby-2.7.5/ruby/2.7.0/gems/libhoney-2.1.0/lib/libhoney/transmission.rb:235
Allocated OBJECT 109 objects of size 4360/2336608 (in bytes) at: /bundle/2021-12-13/ruby-2.7.5/ruby/2.7.0/gems/libhoney-2.1.0/lib/libhoney/response.rb:20
Allocated STRING 60 objects of size 2710/2336608 (in bytes) at: /bundle/2021-12-13/ruby-2.7.5/ruby/2.7.0/gems/excon-0.92.2/lib/excon/response.rb:191
Allocated OBJECT 50 objects of size 2000/2336608 (in bytes) at: /bundle/2021-12-13/ruby-2.7.5/ruby/2.7.0/gems/excon-0.92.2/lib/excon/connection.rb:279

excon appears there, and apologies if the problem is in that gem instead, but since libhoney objects appear above, and I know libhoney uses excon, it felt like the problem might be with libhoney keeping some excon related objects in memory.

I don't have a steps to reproduce, apologies for not making a better issue. I don't have a lot more time to spare, and it seemed like a serious enough issue that you might want to know about it. Happy to provide any other background you might find interesting.

Versions

kentquirk commented 2 years ago

@mrsimo Last year in PR #87, @robbkidd added experimental_transmission to try to address a similar problem. Any chance you could give it a try and let us know if it fixes this for you?

MikeGoldsmith commented 1 year ago

Closing as it's gone stale, please feel free to re-open as required.