Concurrent::Map Performance

[totrash]

* Operating system:                mac
* Ruby implementation:             Ruby 2.7.1
* `concurrent-ruby` version:       1.1.7
* `concurrent-ruby-ext` installed: yes / no
* `concurrent-ruby-edge` used:    no

I tried to use Concurrent::Map instead of Concurrent::Hash because I don't need ordered Hash and I suppose that this is the main difference between them.

Unfortunately, my tests show that Map is generally slower especially on writing

Am I understand it right that Concurrent::Map is created for better performance when You don't need ordered inserts in Hash ?

My tests full code Results Results with concurrent-ruby-ext

One simple test of reading/writing/deleting operations looks like this:

        Benchmark.ips do |bm|
          bm.report "Hash" do
            (1..Concurrent::ThreadSafe::Test::THREADS).map do |i|
              in_thread do
                ITERATIONS.times do |j|
                  key = i * 1000 + j
                  hsh[key] = i
                  hsh[key]
                  hsh.delete(key)
                end
              end
            end.map(&:join)
          end
          bm.report "Map" do
            (1..Concurrent::ThreadSafe::Test::THREADS).map do |i|
              in_thread do
                ITERATIONS.times do |j|
                  key = i * 1000 + j
                  map[key] = i
                  map[key]
                  map.delete(key)
                end
              end
            end.map(&:join)
          end
          bm.compare!
        end
      end

And here are some results:

  WITH CONCURRENCY

------
WRITING/READING/DELETING
-------

Warming up --------------------------------------
                Hash     1.000  i/100ms
                 Map     1.000  i/100ms
Calculating -------------------------------------
                Hash      1.375  (± 0.0%) i/s -      7.000  in   5.102224s
                 Map      0.815  (± 0.0%) i/s -      5.000  in   6.151515s

Comparison:
                Hash:        1.4 i/s
                 Map:        0.8 i/s - 1.69x  (± 0.00) slower

-------------------
WRITING 
-------------------

Warming up --------------------------------------
                Hash     1.000  i/100ms
                 Map     1.000  i/100ms
Calculating -------------------------------------
                Hash      3.184  (± 0.0%) i/s -     16.000  in   5.057282s
                 Map      1.797  (± 0.0%) i/s -     10.000  in   5.609131s

Comparison:
                Hash:        3.2 i/s
                 Map:        1.8 i/s - 1.77x  (± 0.00) slower

-------------------
READING
-------------------

Warming up --------------------------------------
                Hash     1.000  i/100ms
                 Map     1.000  i/100ms
Calculating -------------------------------------
                Hash      2.773  (± 0.0%) i/s -     14.000  in   5.091332s
                 Map      2.200  (± 0.0%) i/s -     11.000  in   5.033331s

Comparison:
                Hash:        2.8 i/s
                 Map:        2.2 i/s - 1.26x  (± 0.00) slower

   WITHOUT CONCURRENCY

-------------------
READING
-------------------

Warming up --------------------------------------
                Hash     1.000  i/100ms
                 Map     1.000  i/100ms
Calculating -------------------------------------
                Hash      2.999  (± 0.0%) i/s -     15.000  in   5.046936s
                 Map      2.419  (± 0.0%) i/s -     13.000  in   5.392514s

Comparison:
                Hash:        3.0 i/s
                 Map:        2.4 i/s - 1.24x  (± 0.00) slower

-------------------
WRITING
-------------------

Warming up --------------------------------------
                Hash     1.000  i/100ms
                 Map     1.000  i/100ms
Calculating -------------------------------------
                Hash      3.092  (± 0.0%) i/s -     16.000  in   5.245244s
                 Map      1.763  (± 0.0%) i/s -      9.000  in   5.225075s

Comparison:
                Hash:        3.1 i/s
                 Map:        1.8 i/s - 1.75x  (± 0.00) slower

-------------------
WRITING/READING/DELETING
-------------------

Warming up --------------------------------------
                Hash     1.000  i/100ms
                 Map     1.000  i/100ms
Calculating -------------------------------------
                Hash      1.506  (± 0.0%) i/s -      8.000  in   5.320253s
                 Map      0.731  (± 0.0%) i/s -      4.000  in   5.557450s

Comparison:
                Hash:        1.5 i/s
                 Map:        0.7 i/s - 2.06x  (± 0.00) slower

With concurrent-ruby-ext installed results are similar.

[kvokka]

the test is wrong map is subclass of Array, so key deletion is a way more complex procedure than key deletion on a hash (feel free to measure it with ruby primitives)

and the fact, that it's only 2 times slower shows a great job done by ruby team!

[totrash]

@kvokka

Thanks for replay. I still dont get it the idea behind those 2 classes. From documentation:

"Map A hash-like object that should have much better performance characteristics, especially under high concurrency, than Concurrent::Hash."

I see a lot of Concurrent::Map usage in some open source project but why should anybody use it if Concurrent::Hash is faster ?

[eregon]

@kvokka What do you mean? There is no Array involved in Concurrent::Map as far as I can see.

@totrash It's a fair question. Currently on CRuby Concurrent::Hash is just ::Hash and Concurrent::Map is a ::Hash wrapped with a lock for some operations (MriMapBackend). That's most likely why Concurrent::Map is slightly slower than Concurrent::Hash on CRuby. We might need to add synchronization to Concurrent::Hash on CRuby too to solve other issues like #970 and #929 and then they would probably have the same performance on CRuby.

Now the point about Concurrent::Map is it enables concurrent/parallel access to the map but for that you need a Ruby implementation without a global lock, so JRuby or TruffleRuby. CRuby never executes Ruby code in parallel (well except Ractors).

So use Concurrent::Hash when you need ordering, and use Concurrent::Map when you don't need ordering. On CRuby the difference is unlikely to be big enough to notice (and will probably change with concurrent-ruby releases), on other Rubies, Concurrent::Map scales and runs in parallel while Concurrent::Has serializes everything by a single lock or contending memory accesses.

[eregon]

With https://github.com/ruby-concurrency/concurrent-ruby/pull/989 there is almost no difference for #[] between Concurrent::Map and Concurrent::Hash.