search

socketry / async-redis

MIT License
83 stars 18 forks source link

Performance specs #8

Closed ioquatix closed 5 years ago

ioquatix commented 5 years ago

@davidor thanks for all your effort digging into https://github.com/socketry/async-redis/pull/7

One thing I've done in the past is include benchmark-ips specs to compare things. While it doesn't give you anything that you can really expect, it does allow you to compare and contrast things if there are changes in the implementation.

For example:

https://github.com/ioquatix/trenni/blob/03740c9e51211ebdf691d1ebdbde53b1d7fd0cac/spec/trenni/parsers_performance_spec.rb#L16-L40

This spec just compares nokogiri with my own parser.

If there is anther implementation of Redis, it might make sense to compare some basic operations and check they perform roughly the same speed, or even just report the speed difference.

If you have time to write such a spec, feel free to include a different implementation of redis as a development dependency.

ioquatix commented 5 years ago

Here is an example output FYI:

Trenni::Native
  #parse_markup
Warming up --------------------------------------
      Large (Trenni)    78.000  i/100ms
    Large (Nokogiri)    30.000  i/100ms
Calculating -------------------------------------
      Large (Trenni)    800.556  (±10.5%) i/s -      3.978k in   5.023464s
    Large (Nokogiri)    334.007  (±10.8%) i/s -      1.650k in   5.014023s

Comparison:
      Large (Trenni):      800.6 i/s
    Large (Nokogiri):      334.0 i/s - 2.40x  slower

So when you are developing if you see a large discrepancy (e.g. different platform or different code) you might choose to investigate further.

ioquatix commented 5 years ago

@k1tsu do you want to take a stab at this?

ioquatix commented 5 years ago

Okay, great now we have a comparison:

Client Performance
Warming up --------------------------------------
         async-redis    38.000  i/100ms
            redis-rb   243.000  i/100ms
Calculating -------------------------------------
         async-redis      7.286k (± 9.7%) i/s -     33.630k in   5.001700s
            redis-rb     17.629k (± 9.2%) i/s -     87.480k in   5.011515s

Comparison:
            redis-rb:    17628.8 i/s
         async-redis:     7286.0 i/s - 2.42x  slower

We should probably investigate why the perf is 2.4x slower. One easy way to do that is to use rbspy.

ioquatix commented 5 years ago

After playing around with the way IO is buffered, I managed to get this micro-benchmark to be about equivalent, which is good.

https://travis-ci.org/socketry/async-redis/jobs/500989051#L308-L309