search

honeycombio / beeline-ruby

Legacy instrumentation for your Ruby apps with Honeycomb.
Apache License 2.0
22 stars 32 forks source link

High memory usage #143

Closed pedrocarrico closed 2 years ago

pedrocarrico commented 3 years ago

Hi!

I'm experiencing a very unusual high memory usage right after installing the honeycomb gem. I have a rails app with ruby 2.7.2 with jemalloc 5.2.1, rails 6.0.3 running on Puma with 2 workers with 5 threads each and I've taken a report with memory_profiler of 15 minutes of usage and the difference is noticeable with and without the honeycomb gem:

With honeycomb Without honeycomb
Total allocated: 950.06 MB (9818855 objects) 510.56 MB (5647235 objects)
Total retained: 46.37 MB (234731 objects) 32.58 MB (241419 objects)

And just a bit of allocated memory by gem that I found interesting:

gem name With honeycomb Without honeycomb
activerecord-6.0.3.4 284.12 MB 182.89 MB
activesupport-6.0.3.4 216.33 MB 155.99 MB
railties-6.0.3.4 72.93 MB 50.50 MB
libhoney-1.18.0 61.65 MB
honeycomb-beeline-2.4.0 19.81 MB
activemodel-6.0.3.4 11.49 MB 8.45 MB

Perhaps some kind of instrumentation on rails might be causing it? Would it be possible to investigate this a bit further?

I'm able to pair program and give you the whole report if needed just can't post it here as it has it has some details of the application I'd rather not expose.

Thank you very much

robbkidd commented 3 years ago

Another issue related to memory-consumption: honeycombio/libhoney-rb#73

robbkidd commented 3 years ago

The memory leak from Timeout in libhoney-rb has been confirmed. The proposed change to fix that is on our backlog for triage.

There could be some objects lingering around from the instrumentations, but I wasn't able to identify them conclusively within the time box we had for investigation as originating in the Beeline or in Rails' ActiveSupport::Notifications instrumentation subsystem. If you have more details about the objects you see being retained, we can arrange for a discrete transfer of that data in the Pollinators Slack. Optionally, there is a Rails 5.2 example app with the Beeline instrumentations enabled that you could explore memory profiling with separate from your own application's business logic.

MikeGoldsmith commented 2 years ago

The alternative transmission is now available https://github.com/honeycombio/libhoney-rb/pull/87 has been merged and we also now support OTLP/HTTP at the Honeycomb API so the OpenTelemetry SDK can be used in place of the Beeline if you'd prefer to use that.

Closing this as there are now two alternatives. Please reopen if you think it's still relevant.