search

phpredis / phpredis

A PHP extension for Redis
Other
9.98k stars 2.14k forks source link

"Connection timed out" exception #2085

Open ilya-levin-lokalise opened 2 years ago

ilya-levin-lokalise commented 2 years ago

Hi everyone! Couldn't find a better place to ask my question. We have a high-load project that utilizes Redis a lot. The problem is - we occasionally are having RedisException being thrown with Connection timed out message, even though the timeout is pretty high (several seconds). We're trying to track down the issue and our monitoring data shows that there are no issues with connection from PHP host to Redis host, however, the PHP application itself is occasionally throws this exception. Long story short - I've downloaded the source of the module (this repo) and was trying to find Connection timed out string or the place where this exception is thrown and failed. I wanted to check what are the conditions for this exception to be thrown. Can someone point me out?

Venorcis commented 2 years ago

We actually have the same issue since about a month now (I'd say since the 5.3.7 release), just random Connection Timeouts happening, no luck in tracking anything down in our servers (we even doubled the capacity of all servers to try and resolve it).

Edit: I actually just checked, and we updated to phpredis 5.3.7 on the 28th of February (sourced from the remi rpm), and right from that date on these timeouts started happening. We'll try downgrading to 5.3.6 to see if that helps.

ilya-levin-lokalise commented 2 years ago

Our project is currently using a pretty old 4.3.0 version of the extension and facing the same issue.

Venorcis commented 2 years ago

We actually have the same issue since about a month now (I'd say since the 5.3.7 release), just random Connection Timeouts happening, no luck in tracking anything down in our servers (we even doubled the capacity of all servers to try and resolve it).

Edit: I actually just checked, and we updated to phpredis 5.3.7 on the 28th of February (sourced from the remi rpm), and right from that date on these timeouts started happening. We'll try downgrading to 5.3.6 to see if that helps.

Actually the downgrade does not help. However we did also upgrade from PHP 8.0 to 8.1 at that same day, so now suspecting that that's related.

deniscsz commented 1 year ago

I have experiencing the same scenario with PHP 8.1.

fwilliamconceicao commented 1 year ago

I'm having this issue with PHP 8.2.

chillidoor commented 1 year ago

We're seeing this issue with PHP 8.1 and 7.3

dellow commented 1 year ago

Also seeing this with 8.2

pablorsk commented 1 year ago

Hello, maybe our context can be helpful.

Only happens in docker (php:8.1-fpm-alpine) when PHP is running under crond. We precisely use Laravel and it happens with php artisan schedule:run.

We have exactly the same problem under the same circumstances:

We are currently in dialogue with DigitalOcean support, since we use the database managed by them.

deniscsz commented 1 year ago

What worked to me: make the tls version explicit in the database connection.

For some reason when I tried use tlsv1.2 in the Redis connection in scheme parameter the issue is gone.

I found this "solution" in this blog post: https://alexvanderbist.com/2021/slow-queues-and-redis-over-tls/

chillidoor commented 1 year ago

In our case, we were already using tls 1.2.

Similar to @pablorsk, we're using the official PHP docker images, and also are using DigitalOcaen's managed Redis database. Increasing the time to >120s hasn't solve the problem for us.

fwilliamconceicao commented 1 year ago

In our case, we were already using tls 1.2.

Similar to @pablorsk, we're using the official PHP docker images, and also are using DigitalOcaen's managed Redis database. Increasing the time to >120s hasn't solve the problem for us.

This doesn't have a relation with TLS 1.2, we've been using it here for 2 years and this issue started triggering randomly in April until now, but now it doesn't happen with frequency.

But at least once per week (on a random day) that happens.

pablorsk commented 1 year ago

We have the same case to @fwilliamconceicao, no related with tls. We try with persistent=1, read_write_timeout=0 and read_write_timeout=-1.

Maybe this information will be helpful. This error only happens when the php is executed by crond. In detail, the structure is: Docker php:8.1-fpm -> supervisord -> crond -> php artisan schedule:run

This PHP fails randomly, between 30 and 180 minutes, on any REDIS.GET. The script does a get and freezes, finally causing a timeout.

pablorsk commented 11 months ago

UPDATE

I just have to try implementing my own redis on our k8s cluster.

fwilliamconceicao commented 11 months ago

UPDATE

  • Not related with supervisord or crond, also fails randomly without those tools.
  • Not related with CPU/Memory resources. I have upgraded the machine to one with two dedicated cores and the problem persists.

I just have to try implementing my own redis on our k8s cluster.

For sure it's not related to any tool, I tried to remove any framework behind it (here we use Laravel + Horizon) and the problem persisted even with a fresh install, I tried with a machine with 4vCPU (dedicated as well) and the problem persist.

It's something related to how the library uses the native PHP resources to communicate with REDIS, anyway, this issue seems to me like it'll be abandoned.

might be in the future I'll fork the library and fix it (like I did with other projects) and keep going.

cyppe commented 6 months ago

Nothing to add more than that I am also seeing the same issue. Only for php artisan schedule:run, and only when it runs in cron.

Works fine when I run command manually.