Add benchmark for `return or =` vs `||=`

[jperville]

The ||= operator is commonly used to implement memoization.

This benchmark shows a much faster alternative to implement memoization: explicit return when the memoized value should be reused, with fallback to setting the memoized value.

$ ruby -v code/general/return-or-set-vs-or-equals.rb
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux-gnu]
Warming up --------------------------------------
                 ||=    12.219k i/100ms
         return or =    16.085k i/100ms
Calculating -------------------------------------
                 ||=    127.190k (± 0.9%) i/s -    647.607k in   5.092082s
         return or =    166.137k (± 0.9%) i/s -    836.420k in   5.034955s

Comparison:
         return or =:   166137.0 i/s
                 ||=:   127190.4 i/s - 1.31x  slower

[ixti]

I'm not quiet sure what this benchmark is trying to show off. It's obvious that ||= doing a bit more than just =, but the purpose is different. What's the point of assigning a constant? If you will use test cases that are closer to real world - difference will be unnoticeable. Comparing = to ||= is IMO even less valuable than comparing Enumerable#each to Enumerable#map.

[jperville]

@ixti we got the following real world case: we are using the json-ld gem and want to frame a graph.

In the following stackprof trace, we spend 26% of the running time in 2 methods, with 12.1% in RDF::URI#value:

$ stackprof tmp/stackprof-cpu-*.dump --text --limit 2
==================================
  Mode: cpu(1000)
  Samples: 107 (0.00% miss rate)
  GC: 11 (10.28%)
==================================
     TOTAL    (pct)     SAMPLES    (pct)     FRAME
        18  (16.8%)          15  (14.0%)     RDF::URI#==
        13  (12.1%)          13  (12.1%)     RDF::URI#value

Here is the code of RDF::URI#value (on github: https://github.com/ruby-rdf/rdf/blob/2.2.9/lib/rdf/model/uri.rb#L798-L806):

$ stackprof tmp/stackprof-cpu-*.dump --text --method 'RDF::URI#value'
RDF::URI#value (/home/julien/RubymineProjects/rdf/lib/rdf/model/uri.rb:798)
  samples:    13 self (12.1%)  /     13 total (12.1%)
  callers:
       9  (   69.2%)  RDF::URI#to_str
       4  (   30.8%)  RDF::URI#to_str
  code:
                                  |   798  |     def value
                                  |   799  |       @value ||= [
                                  |   800  |         ("#{scheme}:" if absolute?),
                                  |   801  |         ("//#{authority}" if authority),
                                  |   802  |         path,
                                  |   803  |         ("?#{query}" if query),
                                  |   804  |         ("##{fragment}" if fragment)
   13   (12.1%) /    13  (12.1%)  |   805  |       ].compact.join("").freeze
                                  |   806  |     end

The expensive build of the default value is only invoked once in our case (out of 10000s of calls to RDF::URI#value).

By adding an early return of the memoized variable if present, the time spent in the method went from 12% to 9% of the total, as show here:

$ stackprof tmp/stackprof-cpu-*.dump --text --method 'RDF::URI#value'
RDF::URI#value (/home/julien/RubymineProjects/rdf/lib/rdf/model/uri.rb:798)
  samples:     9 self (8.7%)  /      9 total (8.7%)
  callers:
       6  (   66.7%)  RDF::URI#to_str
       3  (   33.3%)  RDF::URI#to_str
  code:
                                  |   798  |     def value
    9    (8.7%) /     9   (8.7%)  |   799  |       return @value if @value
                                  |   800  |       @value ||= [

[jperville]

@texpert I have updated my benchmarks to take your comments into accounts, the results are interesting if we are sure that the memoized variable is defined. It seems that it is the defined? test which explains why such a big difference.

$ ruby -v code/general/return-or-set-vs-or-equals.rb
ruby 2.4.1p111 (2017-03-22 revision 58053) [x86_64-linux-gnu]
Warming up --------------------------------------
                 ||=    12.493k i/100ms
return @value if defined?(@value) && @value)
                        11.297k i/100ms
return @value if defined?(@value)
                        11.903k i/100ms
return @value if @value
                        16.584k i/100ms
Calculating -------------------------------------
                 ||=    128.041k (± 2.7%) i/s -    649.636k in   5.077706s
return @value if defined?(@value) && @value)
                        112.480k (± 2.4%) i/s -    564.850k in   5.024635s
return @value if defined?(@value)
                        119.103k (± 2.5%) i/s -    595.150k in   5.000107s
return @value if @value
                        167.953k (± 2.3%) i/s -    845.784k in   5.038625s

Comparison:
return @value if @value:   167953.2 i/s
                 ||=:   128041.4 i/s - 1.31x  slower
return @value if defined?(@value):   119103.0 i/s - 1.41x  slower
return @value if defined?(@value) && @value):   112480.4 i/s - 1.49x  slower

[ixti]

Oh. That's interesting. Although still a bit strange:

class Memoizer
  VALUE = "xxx"

  def initialize
    @value = nil
  end

  def return3
    return @value if defined?(@value)
    @value
  end
end

At bare minimum #return3 will always return you nil. But your real world example actually pretty interesting indeed.

[Arcovion]

In my tests @value || @value = VALUE is fastest

[Arcovion]

I'm guessing it's slightly faster as @value ||= VALUE has to check whether the instance variable exists, and assign it otherwise. @value || ... simply calls @value without checking, giving you a NameError if the variable doesn't exist.

[jperville]

@Arcovion that's good enough for my use-case, since I know that the variable exist. Updating my PR with your solution as the fastest.

[ixti]

First of all, I want to emphasize that ||= and || works ONLY if your memoized value is NON-falsey, so if your expensive computation returns nil or false, then ||= won't work for you at all. Secondly, calling @varname when it was not initialized before will cause ruby warning, the only form you can use it without defined? guard is ||=:

 class X
  def a
    @a ||= 1
  end

  def b
    @b || @b = 2
  end
end

puts X.new.a
puts X.new.b

run that with ruby -w and you will get warning: instance variable @b not initialized

[ixti]

Also, benchmarks are highly affected with X.times. So this is my small take on this benchmarks:

require "benchmark/ips"

class Memoizer
  VALUE  = "some value".freeze
  CYCLES = 10_000

  def initialize
    @v1 = nil
  end

  def v11
    @v1 ||= VALUE
  end

  def v12
    return @v1 if @v1
    @v1 = VALUE
  end

  def v13
    @v1 || @v1 = VALUE
  end

  def v21
    @v2 ||= VALUE
  end

  def v22
    return @v2 if defined?(@v2)
    @v2 = VALUE
  end

  def v23
    return @v2 if instance_variable_defined?(:@v2)
    @v2 = VALUE
  end

  def self.example(name)
    obj = new

    obj.singleton_class.class_eval <<-RUBY
      alias_method :run, :#{name}
      def to_proc; proc { #{Array.new(CYCLES, "run").join(" ; ")} }; end
    RUBY

    obj
  end
end

puts "== v1x"

Benchmark.ips do |x|
  x.report("v11", &Memoizer.example(:v11))
  x.report("v12", &Memoizer.example(:v12))
  x.report("v13", &Memoizer.example(:v13))
  x.compare!
end

puts "== v2x"

Benchmark.ips do |x|
  x.report("v21", &Memoizer.example(:v21))
  x.report("v22", &Memoizer.example(:v22))
  x.report("v23", &Memoizer.example(:v23))
  x.compare!
end

with the above, results are:

                 v13:     3195.8 i/s
                 v12:     3160.5 i/s - same-ish: difference falls within error
                 v11:     1738.9 i/s - 1.84x  slower

                 v21:     1707.4 i/s
                 v22:     1565.7 i/s - 1.09x  slower
                 v23:     1305.5 i/s - 1.31x  slower

[etagwerker]

Closing this because there were comments with valid points that haven't been addressed in years.