Closed Thorakh closed 11 years ago
Odd that this would occur with only 100 connections. Some steps to debug:
redis-benchmark -c 150
to see if this is a PHP issueLemme know what you find. Thanks!
Hi,
we also had this issue on our server. We are running Mage EE 1.11.2.0 and use redis as cache backend for block- and fullpage caches with lots of optimizations on additional block caching.
The problem is not the module itself, the problem is the default redis-configuration. Redis is per default configured to save its database each minute if more than 10000 changes occoured during that time. In our situation, redis stored about 1.5-2GB database every 2-5 minutes causing high io on the server. Though redis spawns another process to save its database, the ioload caused too many connections created by the webserver to redis.
We disabled write of the redis database to hdd completely, as we only store cache data with that redis server. After that we did not get any connections errors anymore (and we don't have that high ioload anymore).
Configuration in redis (usually /etc/redis/redis.conf or /etc/redis/6379.conf or whatever our redis config file is):
save 900 1 save 300 10 save 60 10000
If you don't rely on the storage of your cache data just comment these out.
Even the restart of apache or flushing cached data during high load times on the webserver works fine with this, as redis really gets lots of data changes caused by this.
Best Thomas
Hey thanks a lot for your feedback ! we gonna try this conf asap ...
i will let u know the result !
Thanx again Masmrlar & Colin
Thanks for the info, Thomas. I'm curious how many keys your cache has when warm and how many products and store scopes it has to result in so many changes, if you don't mind sharing that info?
Yes, I think its ok to share that information. We have something about 17000 active products in one website/storeview. As we sell tuning and replacement parts for vehicles, we have a special filter to enable filtering on vehicletypes/manufacturers/models. This filtering cannot be represented by the usual attribute/layered filtering in mage, so it was implemented by an external company.
Now for the stats of redis (just some parts of 'info' in redis-cli):
used_memory:3760460592 used_memory_human:3.50G used_memory_rss:3805249536 used_memory_peak:3760520744 used_memory_peak_human:3.50G mem_fragmentation_ratio:1.01 ...
db0:keys=4,expires=2 db1:keys=43516,expires=20410 db2:keys=247371,expires=243211
db1 is configured as cache backend in mage, db2 as fullpage cache backend in mage. Sometimes data is written to db0, I don't know why
Lots of data changes are caused by
We changes the redis-store 4 days ago and the cache backend works now without any error message like the one Thorakh described.
@Thorakh : Did you change your redis configuration? Any connection failures since then?
Cool. It is helpful to know the issues people come across as your site gets more traffic than mine. :)
I would consider running two separate Redis instances rather than one, though. Reasons:
Thanks again for your help!
Hey pals,
As masmrlar told us, we try to disable the write of the DB on the HDD but no luck here ... we still have the error ... Here is the info for redis :
redis_version:2.4.2 redis_git_sha1:00000000 redis_git_dirty:0 arch_bits:64 multiplexing_api:epoll process_id:25230 uptime_in_seconds:500476 uptime_in_days:5 lru_clock:1483039 used_cpu_sys:3326.40 used_cpu_user:1289.09 used_cpu_sys_children:2.12 used_cpu_user_children:7.15 connected_clients:5 connected_slaves:1 client_longest_output_list:0 client_biggest_input_buf:0 blocked_clients:0 used_memory:495834920 used_memory_human:472.87M used_memory_rss:525275136 used_memory_peak:578483488 used_memory_peak_human:551.68M mem_fragmentation_ratio:1.06 mem_allocator:jemalloc-2.2.1 loading:0 aof_enabled:0 changes_since_last_save:1259268 bgsave_in_progress:0 last_save_time:1335802932 bgrewriteaof_in_progress:0 total_connections_received:736269 total_commands_processed:57398029 expired_keys:16446 evicted_keys:391376 keyspace_hits:54339242 keyspace_misses:3738595 pubsub_channels:0 pubsub_patterns:0 latest_fork_usec:115545 vm_enabled:0 role:master db0:keys=8086,expires=3562
As you can see our cache is quite small for the moment, cause we deactived the cache on category page and product page due to the error trowed.
Am gonna take a look at the phpredis issue list ... i will report here if i find something.
thx again for the help !
Perhaps try updating to the latest Redis, 2.4.2 is pretty old in Redis release terms..
OK i solve the problem,
i just updated the default value for timeout param on the connect method to redis
Cm_Cache_Backend_Redis/lib/Credis/Client.php L-185
public function __construct($host = '127.0.0.1', $port = 6379, $timeout = 2.5)
I put 5 sec for $timeout and no more error throw \o/
I don't know if it's a good solution, what do you think ?
You can set the timeout in the config as well using
Hi,
this also seems to be a problem on our server in rare cases. The connection timeout seems to occour if products are saved in mage backend during "medium" load times like noon or afternoon. At these times, our staff is still working an we get more and more customers on the site.
I've set the timeout to a real high value (50s) and report, whats happening this afternoon.
FA DMD
WA DUD
Hey Colin !
Firstly i wanna thank you for your awesome work on creating this new cache type.
Sometimes have got this error poping, particulary when traffic is high on the website :
An error occurred connecting to Redis.
Trace: prod/.modman/Cm_Cache_Backend_Redis/lib/Credis/Client.php(278): Credis_Client->connect() prod/.modman/Cm_Cache_Backend_Redis/Cm/Cache/Backend/Redis.php(128): Credis_Client->__call('hGet', Array) prod/.modman/Cm_Cache_Backend_Redis/Cm/Cache/Backend/Redis.php(128): Credis_Client->hGet('zc:k:cc4_CORE_C...', 'd') prod/lib/Zend/Cache/Core.php(303): Cm_Cache_Backend_Redis->load('cc4_CORECACHE...', false) prod/app/code/local/Mage/Core/Model/Cache.php(330): Zend_Cache_Core->load('CORE_CACHE_OPTI...')
for only 100 user connected on redis server.
What's your opinion ?
Do you think this is somethink related with server configuration ? Have you ever experimented something like that ?