Performance drops for large values

[minad]

Hi Evan,

I am working on the moneta key/value library which includes a nice benchmarking tool. I compared Dalli and your memcached. For large values the performance of memcached gets really bad and even drops below Dalli.

• Moneta: https://github.com/minad/moneta
• Benchmarks are on Travis: http://travis-ci.org/minad/moneta

Did I make a mistake or is this a know behaviour? I use the binary protocol because I don't want to base64 the keys. Would be nice to hear from you!

Daniel

[evan]

Not seeing it over here; something mysterious is going on.

Darwin winter.local 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin12]

Ruby 1.9.3p194
Loaded memcached 1.5.0
Loaded remix-stash 1.1.3
Loaded memcache-client 1.8.5
Loaded kgio 2.7.4
Loaded dalli 2.6.0
PID is 10593
Loops is 10000
Stack depth is 0
Small value size is: 19 bytes
Large value size is: 32861 bytes
No matching processes belonging to you were found
                                           user     system      total        real
set-large: dalli:bin                   1.830000   1.430000   3.260000 (  4.816254)
set-large: libm:ascii                  0.340000   0.900000   1.240000 (  2.804563)
set-large: libm:ascii:pipeline         0.320000   0.720000   1.040000 (  1.046644)
set-large: libm:ascii:udp              0.240000   0.840000   1.080000 (  1.927089)
set-large: libm:bin                    0.290000   0.950000   1.240000 (  2.964995)
set-large: libm:bin:buffer             0.340000   1.030000   1.370000 (  1.856447)
set-large: mclient:ascii               3.090000   1.390000   4.480000 (  4.779375)
set-large: stash:bin                   4.350000   1.080000   5.430000 (  6.501960)

get-large: dalli:bin                   1.290000   0.620000   1.910000 (  2.832102)
get-large: libm:ascii                  0.210000   0.340000   0.550000 (  1.297549)
get-large: libm:ascii:pipeline         0.260000   0.480000   0.740000 (  1.676853)
get-large: libm:ascii:udp              0.150000   0.180000   0.330000 (  0.639593)
get-large: libm:bin                    0.280000   0.670000   0.950000 (  1.928862)
get-large: libm:bin:buffer             0.280000   0.760000   1.040000 (  1.928133)
get-large: mclient:ascii               2.950000   0.730000   3.680000 (  3.715739)
get-large: stash:bin                   0.800000   0.500000   1.300000 (  2.181267)

Darwin winter.local 12.2.0 Darwin Kernel Version 12.2.0: Sat Aug 25 00:48:52 PDT 2012; root:xnu-2050.18.24~1/RELEASE_X86_64 x86_64
ruby 1.9.3p194 (2012-04-20 revision 35410) [x86_64-darwin12]

Ruby 1.9.3p194
Loaded memcached 1.5.0
Loaded remix-stash 1.1.3
Loaded memcache-client 1.8.5
Loaded kgio 2.7.4
Loaded dalli 2.6.0
PID is 12268
Loops is 1000
Stack depth is 0
Small value size is: 19 bytes
Large value size is: 524382 bytes
No matching processes belonging to you were found
                                           user     system      total        real
set-large: dalli:bin                   1.250000   2.880000   4.130000 (  7.224394)
set-large: libm:ascii                  0.290000   1.330000   1.620000 (  1.822667)
set-large: libm:ascii:pipeline         0.260000   1.230000   1.490000 (  1.524888)
set-large: libm:ascii:udp              0.200000   1.160000   1.360000 (  2.092535)
set-large: libm:bin                    0.260000   1.290000   1.550000 (  1.761770)
set-large: libm:bin:buffer             0.290000   1.390000   1.680000 (  1.872404)
set-large: mclient:ascii               0.740000   1.230000   1.970000 (  2.093977)
set-large: stash:bin                   5.840000   2.690000   8.530000 ( 11.136886)

get-large: dalli:bin                   0.130000   0.070000   0.200000 (  0.288951)
get-large: libm:ascii                  0.020000   0.030000   0.050000 (  0.125926)
get-large: libm:ascii:pipeline         0.020000   0.050000   0.070000 (  0.167231)
get-large: libm:ascii:udp              0.020000   0.020000   0.040000 (  0.107839)
get-large: libm:bin                    0.030000   0.070000   0.100000 (  0.191549)
get-large: libm:bin:buffer             0.030000   0.070000   0.100000 (  0.179888)
get-large: mclient:ascii               0.290000   0.070000   0.360000 (  0.361704)
get-large: stash:bin                   0.090000   0.060000   0.150000 (  0.257239)

It looks like you're doing the right thing with turning off Ruby marshal, but even with marshal enabled I don't see the degradation.

PS. I would recommend using more than one server instance for benchmarking so as to exercise the hash function. Some clients short-circuit the single-server case and some do not.

PPS. Note that the binary protocol is not faster. The implementation of the binary protocol is more complex.

[minad]

I will investigate this further. Can you keep this open until I resolved it?

[evan]

OK

[minad]

Hi Evan, I also investigated this a bit further. You can look at the moneta benchmarks at http://travis-ci.org/minad/moneta for different value sizes and key/value size distributions.

[evan]

Dunno; if you can reproduce in the Gem's own benchmarks I will fix.