Single worker thread is too slow

[GoogleCodeExporter]

What version of the product are you using? On what operating system?

We're running on Ubuntu/Spring. 

Tell me more...

Using JProfiler, i was able to see a significant performance bottleneck via the 
single worker thread. If I create 100 clients and multiplex between them 
randomly, I am able to get 100x performance with the worker (maybe io?) threads 
evenly 50% filled.

It's my intention to open the source code and make the worker (io?) thread a 
configurable setting.

Original issue reported on code.google.com by smye...@gmail.com on 25 Jan 2012 at 11:56

[GoogleCodeExporter]

That would be a major internal change in Spymemcached. Can you propose how you 
would go about doing this? Also, I can get 30k-50k ops/sec with the current 
implementation. What kind of application do you have that needs significantly 
more throughput than this? 

Original comment by mikewie...@gmail.com on 26 Jan 2012 at 12:08

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

interface SelectionStrategy<T> {
   T select();
}

class BalancedMemcachedClient implements MemcachedClientIF {

 ...

   public Object get(String key) {
       return selectionStrategy.select().get(key);
   }

 ...

}

We have 16 memcached (couchbase?) servers (4 physical and 12 virtual) and our 
java app is copied 90 times across 3 racks. 

We're seeing most of our app waiting on memcached operations (with the IO 
thread completely full). By adding 20 backing memcached clients, our app is now 
waiting on database and/or locks. (as seen by JProfiler thread-view). The IO 
thread is adequately used, and not full.

We (my operations team + myself) are not experts. I do not know the ops/sec 
that i receive before/after, though I do not believe it to be as high as 
50k/sec. Membase's web GUI indicates that we have about 80k/sec operations for 
the cluster.

Original comment by smye...@gmail.com on 26 Jan 2012 at 1:34

[GoogleCodeExporter]

Do you have a suggestion on a way to measure the operations per second of a 
specific MemcachedClient? This seems like a good way to ensure our assumptions 
are correct.

Original comment by smye...@gmail.com on 26 Jan 2012 at 1:36

[GoogleCodeExporter]

I used to write performance tests with Spymemcached an tehe way I did it then 
was to do something like this

long st = System.currentTimeMillis();
client.set(...).get()
long latency = System.currentTimeMillis() - st;

You might want to try doing things asynchronously though so you might have to 
mix you logic into the callbacks. Take a look at the code in MemcachedClient 
and override so of the functions there. It shouldn't be too hard.

Also, we are open to accepting any changes that enhance the performance of 
Spymemcached. This one sounds like it could be a big one so just check in with 
us on what your doing so that it has the best chance of getting merged to the 
project. When you get a little bit further I will let you know how to submit it 
for review.

Original comment by mikewie...@gmail.com on 28 Jan 2012 at 7:32

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

maaliqadoumi: I don't have enough data to say why connecting is slower.  I saw 
your earlier note too.  It's not appropriate to post that to a comment on this 
issue though, and please refrain from doing that.  If you have data about this 
issue, please comment on it here.  If not, please take it to the mailing list.

Suggestion: gather some info by turning up logging if you want help in what's 
going on.

Original comment by ingen...@gmail.com on 24 Jul 2013 at 7:36