Possible process-level memory leak

[anaice]

Hello everyone, everything good?

I have a Rails 6.1 (ruby 2.7.7) application that was running cable_ready 4.3. After updating some gems (including cable_ready 5.0.5) we have experienced a gradual increase in memory, suggesting that it is a memory leak.

I took some samples from the application heap (15min, 30min and 60 min after startup). Then I ran a heapy diff (https://github.com/zombocom/heapy) of the three samples. In the list of retained objects, I found operation_builder.rb:26, operation_builder.rb:21 as well as operation_builder.rb:9

generated diff - top retained objects

>heapy diff sample1.heap sample2.heap sample3.heap | head -n 30

Retained IMEMO 10611 objects of size 677904/5273042 (in bytes) at: /usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb:26
Retained DATA 5248 objects of size 419840/5273042 (in bytes) at: /usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb:26
(...)
Retained DATA 153 objects of size 12240/5273042 (in bytes) at: /usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb:9

Heap Profiler Report

Report

```sh Total: 371.80 MB (2388011 objects) memory by gem ----------------------------------- 91.47 MB bootsnap-1.18.3 54.54 MB cable_ready-5.0.5 (...) memory by file ----------------------------------- 86.79 MB bootsnap-1.18.3/lib/bootsnap/compile_cache/iseq.rb 54.06 MB cable_ready-5.0.5/lib/cable_ready/operation_builder.rb (...) memory by location ----------------------------------- 85.72 MB bootsnap-1.18.3/lib/bootsnap/compile_cache/iseq.rb:53 52.44 MB cable_ready-5.0.5/lib/cable_ready/operation_builder.rb:26 (...) ```

Sheap

I tried to get memory addresses related to this files and then using sheap (https://github.com/jhawthorn/sheap) to reach HeapObject details. Then I found some hints related to singleton_class.defined_method:

Sheap

```ruby => [[{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>340}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>340}, {:type=>"DATA", :imemo_type=>nil, :struct=>"proc", :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>340}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>340}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>340}, {:type=>"DATA", :imemo_type=>nil, :struct=>"proc", :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>340}, {:type=>"IMEMO", :imemo_type=>"env", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"lambda", :line=>26, :generation=>340}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}, {:type=>"DATA", :imemo_type=>nil, :struct=>"proc", :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}, {:type=>"DATA", :imemo_type=>nil, :struct=>"proc", :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"env", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"lambda", :line=>26, :generation=>218}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}, {:type=>"DATA", :imemo_type=>nil, :struct=>"proc", :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}, {:type=>"DATA", :imemo_type=>nil, :struct=>"proc", :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"env", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"lambda", :line=>26, :generation=>218}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}], [{:type=>"CLASS", :imemo_type=>nil, :struct=>nil, :name=>"CableReady::Channel", :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"singleton_class", :line=>26, :generation=>218}, {:type=>"IMEMO", :imemo_type=>"ment", :struct=>nil, :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}, {:type=>"DATA", :imemo_type=>nil, :struct=>"proc", :name=>nil, :file=>"/usr/local/bundle/ruby/2.7.0/gems/cable_ready-5.0.5/lib/cable_ready/operation_builder.rb", :method=>"define_method", :line=>26, :generation=>218}]] ```

Debugging possible leak

I decided to debug the application up to the cable ready level. I ended up finding a pattern where CableReady::Config starts accumulating references to CableReady::Channel instances.

Channel instantiation mechanism:

There are references to channels both at the process level (CableReady::Config - @observer_peers) and at the thread level (CableReady::Channels - @channels)

However, when the process of clearing the channels begins, it is only done at the thread level (@channels), waiting for it to eventually be released at the process level (via ObjectSpace.define_finalizer). However, since CableReady::Config remains pointing to the channel in @observer_peers, I believe the channel is never garbage collected:

I tested a possible solution and will make a PR.

[julianrubisch]

this is very detailed, thank you! I'll let you know when I get to tackling this

[julianrubisch]

ah, just saw your PR. sorry, will go and inspect it