Show retained difference

[dblock]

tl;dr How do I show retained difference in compare!?

Coming from https://github.com/ruby-grape/grape/pull/2102#issuecomment-693823956, the amount of memory allocated is the same, but we are retaining many more objects. Want to highlight that since we're trying to track a leak.

[AlexWayfer]

Can you please provide a simpler example without external libraries, like Grape?

[dblock]

require 'set'
require 'benchmark/memory'

class Retainer
  attr_reader :setup

  def initialize
    @setup = []
  end

  def retain!
    @setup << { x: 1 }
  end
end

r = Retainer.new

setup_ar = r.instance_variable_get(:@setup).to_a
setup_set = r.instance_variable_get(:@setup).to_set

Benchmark.memory do |b|
  calls = 10001

  b.report('using Array') do
    r.instance_variable_set(:@setup, setup_ar)
    calls.times { r.retain! }
  end

  b.report('using Set') do
    r.instance_variable_set(:@setup, setup_set)
    calls.times { r.retain! }
  end

  b.compare!
end

The output is:

Calculating -------------------------------------
         using Array     1.680M memsize (     1.680M retained)
                        10.001k objects (    10.001k retained)
                         0.000  strings (     0.000  retained)
           using Set     1.680M memsize (   168.000  retained)
                        10.001k objects (     1.000  retained)
                         0.000  strings (     0.000  retained)

Comparison:
         using Array:    1680168 allocated
           using Set:    1680168 allocated - same

Desired output to include the retained difference of 10K objects vs. 1 object.

[AlexWayfer]

1. You forgot to add require 'set', otherwise you'll get:

Traceback (most recent call last):
benchmark-memory.rb:20:in `<main>': undefined method `to_set' for []:Array (NoMethodError)
Did you mean?  to_s

2. What version of Ruby do you use? I'm using the latest, 2.7.1, and here my results:

Calculating -------------------------------------
         using Array     1.680M memsize (     1.680M retained)
                        10.001k objects (    10.001k retained)
                         0.000  strings (     0.000  retained)
           using Set     1.681M memsize (   552.000  retained)
                        10.003k objects (     3.000  retained)
                         0.000  strings (     0.000  retained)

Comparison:
         using Array:    1680168 allocated
           using Set:    1680552 allocated - 1.00x more

It's still not "10K objects vs. 1 object.", but a bit different from yours.

Now I'm going to fix bench, I guess there is a fundamental error, in approach.

[AlexWayfer]

OK, there is no fundamental errors in approach (instance_variable_get and related confused me), but my second assumption was that Garbage Collector just doesn't run in time. So, if you'd add GC.start to the end of both b.report — there will be such results:

Calculating -------------------------------------
         using Array     1.680M memsize (     1.680M retained)
                        10.001k objects (    10.001k retained)
                         0.000  strings (     0.000  retained)
           using Set   552.000  memsize (   552.000  retained)
                         3.000  objects (     3.000  retained)
                         0.000  strings (     0.000  retained)

Comparison:
           using Set:        552 allocated
         using Array:    1680168 allocated - 3043.78x more

[dblock]

I see what's going on, thanks. The output is clear in terms of results with GC.start. I guess I was assuming that it was built in, should it be to report accurate numbers?

[AlexWayfer]

I guess I was assuming that it was built in, should it be to report accurate numbers?

I don't know, I guess it's up to MRI. If GC doesn't run here (even if you'd replace GC.start with sleep 10, for example) — it can not run in real-life application. So… let's leave it up to benchmark-memory's collaborators: if they want to make this benchmarks more clear and theoretical or more practical and closer to a real-life.

[dblock]

Closing in favor of the GC question, https://github.com/michaelherold/benchmark-memory/issues/12.

[dblock]

Thanks @AlexWayfer for helping me work through this.

[AlexWayfer]

Closing in favor of the GC question, #12.

I don't think it's different issues, like we could leave this one, but… as you wish.

Thanks @AlexWayfer for helping me work through this.

You're welcome. I'm interesting in correct benchmarks.