search

kohana / cache

Cache library for Kohana 3
76 stars 79 forks source link

performance degradation on big memcache instances #85

Open Genda1ph opened 8 years ago

Genda1ph commented 8 years ago

Hello. We're using quite big memcached instance for our application - 4GB (8 threads, large pages enabled), which, apparently, causes whole application to significantly slow down. I see gradual rise of Load Average from 2-5 to 20-30 when we pass 512MB cache size mark. If we purge the cache, LA drops back to normal within a couple of minutes. Things got a little bit better after we switched from TCP socket to UNIX socket. And right now we've limited this memcached instance to 512M and it looks like LA did rise a bit after cache usage passed 256M mark, but it's insignificant. get/set rates are around 400/160-175 req/s, but this shouldn't be too much - this is memcached after all.

Where should we start looking into this issue?

We use Kohana 3.3.2 running on Debian 7 with PHP 5.4.36 and memcache 1.4.13 (via memcache driver).

enov commented 8 years ago

Thanks for reporting this @Genda1ph. Not sure what to reply, though. You probably need to use a profiling tool to see where we have a bottleneck in the Kohana code, if any.

Genda1ph commented 8 years ago

Thing is, profiling doesn't help due to sheer number of requests. I tried attaching via strace to fpm aaand no waits seem to happen.

After some more research I can say that cache plugin is not the culprit, but mcache is. Moreover, issue lies not with plugin itself, but, most likely, with memcached and its LRU mechanism - It appears that the larger slab is, the more expensive eviction in that slab is (probably, memcahed has to loop through whole slab, looking for oldest timestamp).

I'm not that knowledgeable on how FPM works, but here's my hypothesis: while php is waiting for something to be evicted from memcached it parks the requests' thread and works with another request, which leads to huge TTFB and rise of LA (which, under the hood, is thread queue length). So, when I reduced instance size to 128M and tweaked chunk sizes a bit, looking for LRU became inexpensive, even though number of evictions doubled-tripled (~100 evictions/sec, up from 30-50) I suggest mentioning this somewhere in the docs and closing the issue, since our TTFB dwindled from ~2s to ~600ms.

tomazov commented 4 years ago

use it $this->cache = Cache::instance(); in place $this->cache = new Memcache();

public function __construct($id = NULL, $group = NULL){
    $this->cache = Cache::instance($group);
    $this->id = $id;
}

and fix

public function set($tables, $sql, $result, $lifetime){
    ...
    $this->cache->set($hash, $this->condense($result), $lifetime);
    ...