OOM command not allowed when used memory > 'maxmemory'

[JacobDorman]

Happy to move to stackoverflow if this is a "support question" however i've trawled through most of the questions there and not come up with a decent solution.

---

Redis is exceeding it's maxmemory which is preventing cache from being cleaned.

Apparently this is intended behaviour:

So we continuously cross the boundaries of the memory limit, by going over it, and then by evicting keys to return back under the limits. If a command results in a lot of memory being used (like a big set intersection stored into a new key) for some time the memory limit can be surpassed by a noticeable amount. http://redis.io/topics/lru-cache

but this leaves the server in a non-functional state, complaining of cache and indexes being out of date and not able to clean or reindex.

OOM command not allowed when used memory > 'maxmemory /srv/www/XXXXX/releases/20160728173201/.modman/Cm_Cache_Backend_Redis/Cm/Cache/Backend/Redis.php(211)

This happens after a large traffic spike. After which most of the admin is inaccessible and the server is swapping memory to disk (possibly causally related?)

Of course there are multiple ways of manually fixing this, however i'm looking for a way to avoid or at least recover from this state. Upgrading the server memory is an obvious choice, but solving the issue seems a better option as the site grows.

• Redis server v=3.0.7
• phpredis Version => 2.2.7
• Transparent Huge Pages Disabled
• TCP Backlog set to 65535
• vm.overcommit_memory=1
• Background saving disabled anyway
• Connected by Unix Socket
• Running provided Redis Garbage Collection twice a day
• Maxmemory was set to 1048576000 (1Gb) however will try reducing this by half overnight

free -m (during)

             total       used       free     shared    buffers     cached
Mem:          2001       1757        244         53         17        162
-/+ buffers/cache:       1578        423
Swap:          999        116        883

Magento Config

    <cache>
      <backend>Cm_Cache_Backend_Redis</backend>
      <backend_options>
        <server>/var/run/redis/redis.sock</server><!-- or absolute path to unix socket -->
        <port>0</port>
        <persistent>magento</persistent> <!-- Specify unique string to enable persistent connections. E.g.: sess-db0; bugs with phpredis and php-fpm are known: https://github.com/nicolasff/phpredis/issues/70 -->
        <database>0</database> <!-- Redis database number; protection against accidental data loss is improved by not sharing databases -->
        <password></password> <!-- Specify if your Redis server requires authentication -->
        <force_standalone>0</force_standalone>  <!-- 0 for phpredis, 1 for standalone PHP -->
        <connect_retries>1</connect_retries>    <!-- Reduces errors due to random connection failures; a value of 1 will not retry after the first failure -->
        <read_timeout>10</read_timeout>         <!-- Set read timeout duration; phpredis does not currently support setting read timeouts -->
        <automatic_cleaning_factor>0</automatic_cleaning_factor> <!-- Disabled by default -->
        <compress_data>0</compress_data>  <!-- 0-9 for compression level, recommended: 0 or 1 -->
        <compress_tags>0</compress_tags>  <!-- 0-9 for compression level, recommended: 0 or 1 -->
        <compress_threshold>20480</compress_threshold>  <!-- Strings below this size will not be compressed -->
        <compression_lib>gzip</compression_lib> <!-- Supports gzip, lzf, lz4 (as l4z) and snappy -->
        <use_lua>0</use_lua> <!-- Set to 1 if Lua scripts should be used for some operations -->
      </backend_options>
    </cache>

 # Memory
used_memory:1048493528
used_memory_human:999.92M
used_memory_rss:1060605952
used_memory_peak:1172501816
used_memory_peak_human:1.09G
used_memory_lua:36864
mem_fragmentation_ratio:1.01
mem_allocator:jemalloc-3.6.0

Output from redis-cli INFO ALL

Output From stats.php

[elfeffe]

Did you find a solution?

[colinmollenhour]

Check out this patch which will significantly reduce cache memory usage from layout XML which is what appears to be eating almost your entire cache. https://gist.github.com/colinmollenhour/14ed23d029e5b991a3e31ad1fe0841d4

[stormearthfire]

@JacobDorman @colinmollenhour

Hi guys, did you perhaps managed to solve this issue? Like Jacob, I've went through all materials online in trying to solve it, but with no luck. I used suggested patch, it does prolong the lifetime of the website, however, still at some point it breaks down.

used_memory:28800640
used_memory_human:27.47M
used_memory_rss:46886912
used_memory_peak:1586815696
used_memory_peak_human:1.48G
used_memory_lua:36864
mem_fragmentation_ratio:1.63
mem_allocator:jemalloc-3.6.0

We observe it through the day, it slowly rises up to about 1.5G and then breaks on this line:

602 $this->_redis->exec();

and since it's not part of the try/catch block it kills the whole site.

[colinmollenhour]

LMSOTFY: https://stackoverflow.com/a/31600531/187780 😁