search

colinmollenhour / Cm_Cache_Backend_Redis

A Zend_Cache backend for Redis with full support for tags (works great with Magento)
Other
390 stars 142 forks source link

READONLY You can't write against a read only replica. #155

Open Genaker opened 4 years ago

Genaker commented 4 years ago

[2020-04-01 01:25:38] main.CRITICAL: READONLY You can't write against a read only replica. {"report_id":"91c1378c517f87a1ffc8455f1ababbe16d709494e48087c35b46fb92b0f70f51","exception":"[object] (CredisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1157, RedisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1119)"} [] [2020-04-01 01:25:38] main.CRITICAL: READONLY You can't write against a read only replica. {"report_id":"91c1378c517f87a1ffc8455f1ababbe16d709494e48087c35b46fb92b0f70f51","exception":"[object] (CredisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1157, RedisException(code: 0): READONLY You can't write against a read only replica. at /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1119)"} []

Magento configuration

backend_options' => [ 'server' => 'redis.0wfhud.ng.0001.use2.cache.amazonaws.com', 'database' => '0', 'port' => '6379', 'password' => '', 'persistent' => 'redis', 'compress_data' => '9', 'connect_retries' => 5, 'compression_lib' => 'gzip', 'load_from_slave' => 'tcp://redis-ro.0wfhud.ng.0001.use2.cache.amazonaws.com:6379', 'retry_reads_on_master' => '1', ]

colinmollenhour commented 4 years ago

Thanks for the report, can you provide a fuller stack trace?

Genaker commented 4 years ago

{"0":"READONLY You can't write against a read only replica.","1":"#1 Cm\RedisSession\Handler->read() called at [vendor\/magento\/framework\/Session\/SaveHandler\/Redis.php:97]\n#2 Magento\Framework\Session\SaveHandler\Redis->read() called at [vendor\/magento\/framework\/Session\/SaveHandler.php:138]\n#3 Magento\Framework\Session\SaveHandler->callSafely() called at [vendor\/magento\/framework\/Session\/SaveHandler.php:83]\n#4 Magento\Framework\Session\SaveHandler->read()\n#5 session_start() called at [vendor\/magento\/framework\/Session\/SessionManager.php:206]\n#6 Magento\Framework\Session\SessionManager->start() called at [vendor\/magento\/framework\/Interception\/Interceptor.php:58]\n#7 Magento\Framework\Session\Generic\Interceptor->callParent() called at [vendor\/magento\/framework\/Interception\/Interceptor.php:138]\n#8 Magento\Framework\Session\Generic\Interceptor->Magento\Framework\Interception\{closure}() called at [vendor\/magento\/framework\/Interception\/Interceptor.php:153]\n#9 Magento\Framework\Session\Generic\Interceptor->callPlugins() called at [generated\/code\/Magento\/Framework\/Session\/Generic\/Interceptor.php:26]\n#10 Magento\Framework\Session\Generic\Interceptor->start() called at [vendor\/magento\/framework\/Session\/SessionManager.php:140]\n#11 Magento\Framework\Session\SessionManager->construct() called at [generated\/code\/Magento\/Framework\/Session\/Generic\/Interceptor.php:14]\n#12 Magento\Framework\Session\Generic\Interceptor->construct() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/AbstractFactory.php:121]\n#13 Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:108]\n#14 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:150]\n#15 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:79]\n#16 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:150]\n#17 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:79]\n#18 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:150]\n#19 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:79]\n#20 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:150]\n#21 Magento\Framework\ObjectManager\Factory\Compiled->get() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:125]\n#22 Magento\Framework\ObjectManager\Factory\Compiled->parseArray() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:86]\n#23 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor\/magento\/framework\/ObjectManager\/ObjectManager.php:70]\n#24 Magento\Framework\ObjectManager\ObjectManager->get() called at [vendor\/magento\/framework\/App\/FrontController.php:71]\n#25 Magento\Framework\App\FrontController->construct() called at [generated\/code\/Magento\/Framework\/App\/FrontController\/Interceptor.php:14]\n#26 Magento\Framework\App\FrontController\Interceptor->construct() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/AbstractFactory.php:121]\n#27 Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject() called at [vendor\/magento\/framework\/ObjectManager\/Factory\/Compiled.php:108]\n#28 Magento\Framework\ObjectManager\Factory\Compiled->create() called at [vendor\/magento\/framework\/ObjectManager\/ObjectManager.php:70]\n#29 Magento\Framework\ObjectManager\ObjectManager->get() called at [vendor\/magento\/framework\/App\/Http.php:115]\n#30 Magento\Framework\App\Http->launch() called at [vendor\/magento\/framework\/App\/Bootstrap.php:261]\n#31 Magento\Framework\App\Bootstrap->run() called at [pub\/index.php:40]\n","url":"\/test-saas-product.html?_tideways%5Bmethod%5D=&_tideways%5Btime%5D=1585705200&_tideways%5Buser%5D=19900&_tideways%5Bhash%5D=d070227bd9cdeebe24698917bce7b91b7405cbd0c13f4656247a64c6d41e3ccb&_tideways_ref=d8b81646-2923-4235-ac80-ea659fb68d7a","script_name":"\/index.php","report_id":"91c1378c517f87a1ffc8455f1ababbe16d709494e48087c35b46fb92b0f70f51"}

What's interesting sessions don't use slaves. Sessions writes to the different Redis

'session' => [ 'save' => 'redis', 'redis' => [ 'host' => '152.14.20.80', 'port' => '6379', 'password' => '', 'timeout' => '2.5', 'persistent_identifier' => 'redis1', 'database' => '2', 'compression_threshold' => '2048', 'compression_library' => 'gzip', 'log_level' => '1', 'max_concurrency' => '6', 'break_after_frontend' => '5', 'break_after_adminhtml' => '30', 'first_lifetime' => '600', 'bot_first_lifetime' => '60', 'bot_lifetime' => '7200', 'disable_locking' => '1', 'min_lifetime' => '60', 'max_lifetime' => '2592000', 'sentinel_master' => '', 'sentinel_servers' => '', 'sentinel_connect_retries' => '5', 'sentinel_verify_master' => '0' ] ],

It is realy rarely issue but I'm having it each day several times.

As I can see from the repost it tries to write the session to the read-only slave DB it uses cache Redis somehow

Genaker commented 4 years ago

It is my full config:

'session' => [ 'save' => 'redis', 'redis' => [ 'host' => '152.14.20.80', 'port' => '6379', 'password' => '', 'timeout' => '2.5', 'persistent_identifier' => 'redis1', 'database' => '2', 'compression_threshold' => '2048', 'compression_library' => 'gzip', 'log_level' => '1', 'max_concurrency' => '6', 'break_after_frontend' => '5', 'break_after_adminhtml' => '30', 'first_lifetime' => '600', 'bot_first_lifetime' => '60', 'bot_lifetime' => '7200', 'disable_locking' => '1', 'min_lifetime' => '60', 'max_lifetime' => '2592000', 'sentinel_master' => '', 'sentinel_servers' => '', 'sentinel_connect_retries' => '5', 'sentinel_verify_master' => '0' ] ], 'cache' => [ 'frontend' => [ 'default' => [ 'idprefix' => 'def', 'backend' => 'Cm_Cache_Backend_Redis', 'backend_options' => [ 'server' => 'redis.0wfhud.ng.0001.use2.cache.amazonaws.com', 'database' => '0', 'port' => '6379', 'password' => '', 'persistent' => 'redis', 'compress_data' => '9', 'connect_retries' => 5, 'compression_lib' => 'gzip', 'load_from_slave' => 'tcp://redis-ro.0wfhud.ng.0001.use2.cache.amazonaws.com:6379', 'retry_reads_on_master' => '1', ] ], 'page_cache' => [ 'idprefix' => 'def', 'backend' => 'Cm_Cache_Backend_Redis', 'backend_options' => [ 'server' => '152.14.20.80', 'database' => '1', 'port' => '6379', 'password' => '', 'persistent' => 'redis2', 'compress_data' => '0', 'compression_lib' => '' ] ] ] ],

kanevbg commented 4 years ago

Experiencing the same issue running on Redis sentinel of 3 nodes.

In my observations error happens every day after full cache flush when a lot of data is present (trough Magento's PHP API, cant say if flushed from Redis directly). Types of errors observed:

Errors are not the same each day, however they are the same after flush, eg. there is only one kind of errors after the given flush for less than a minute, than starts functioning normally.

colinmollenhour commented 4 years ago

Are you both using the Magento patch or Magento LTS which has a fix for the config cache stampeding issue? Without this patch flushing the cache on a site with moderate traffic can easily cause the cache backend to get overwhelmed. Also I recommend using a cache refresh rather than flush so that the config cache gets atomically updated rather than clobbered and rebuilt by the next request hitting a cold cache.

kanevbg commented 4 years ago

My project has Mage_Core_Model_Config class taken from the openmage lts (from master or last release) project, so yes?

This morning all of the three errors had happened after the cache flush (all of the three mentioned types). I have cache flush configured at 5:05 am trough crontab and shell script. From 5:05:07 to 5:05:33 Redis related errors are logged in Magento reports. I will implement cache clearance trough mass refresh to see if this makes difference, however if this do it still seems something is not in place.

EDIT: By the way these two has significant differences, one of which is the useCache flag. Which version is correct:

colinmollenhour commented 4 years ago

I didn't do a full diff, can you point out the difference more precisely?

Here is the atomic ache update I was referring to: https://gist.github.com/colinmollenhour/2715268 I think n98-magerun also has this functionality built in.

kanevbg commented 4 years ago

See the diff here: Report.zip

colinmollenhour commented 4 years ago

Ahh, thanks. Inspecting the code I think that shouldn't be a material difference because the value is declared as false by default and if it is set to true that code in loadModulesCache isn't evaluated anyway.

Genaker commented 4 years ago

I have found the issue

If Redis can't remove keys according to the policy, or if the policy is

set to 'noeviction', Redis will start to reply with errors to commands

that would use more memory, like SET, LPUSH, and so on, and will continue

to reply to read-only commands like GET.

it happens when Redis doesn't have enough memory however looks like Read-only error

Fatal error: Uncaught RedisException: READONLY You can't write against a read only replica. in /var/www/html/magento/vendor/colinmollenhour/credis/Client.php:1119 Stack trace: #0 /var/www/html/magento/vendor/colinmollenhour/credis/Client.php(1119): Redis->exec() #1 /var/www/html/magento/vendor/colinmollenhour/cache-backend-redis/Cm/Cache/Backend/Redis.php(629): Credis_Client->call('exec', Array) #2 /var/www/html/magento/vendor/magento/zendframework1/library/Zend/Cache/Core.php(515): Cm_Cache_Backend_Redis->save('a:1532:{s:11:"p...', 'def_GLOBALDIC...', Array, NULL) #3 /var/www/html/magento/vendor/magento/framework/Cache/Core.php(74): Zend_Cache_Core->save('a:1532:{s:11:"p...', 'def_GLOBAL__DIC...', Array, NULL, 8) #4 /var/www/html/magento/vendor/magento/framework/Cache/Frontend/Adapter/Zend.php(63): Magento\Framework\Cache\Core->save('a:1532:{s:11:"p...', 'GLOBAL::DICONFI...', Array, NULL) #5 /var/www/html/magento/vendor/magento/framework/Cache/Frontend/Decorator/Bare.php(75): Magento\Framework\Cache\Frontend\Adapter\ in /var/www/html/magento/vendor/colinmollenhour/credis/Client.php on line 1157