search

yiisoft / yii2-redis

Yii 2 Redis extension.
http://www.yiiframework.com
BSD 3-Clause "New" or "Revised" License
452 stars 183 forks source link

Can’t release Redis Mutex Lock #248

Closed siggelfredo closed 1 year ago

siggelfredo commented 2 years ago

What steps will reproduce the problem?

We have an issue during our Deployments regarding the Azure Cache for Redis. Within our Kubernetes Cluster we trigger a migration job which acquires a mutex lock and releases this lock after the job is finished.

This is the library we use to connect to redis: https://github.com/yiisoft/yii2-redis And here is the function that should release the lock: https://github.com/yiisoft/yii2-redis/blob/master/src/Mutex.php#L131

It seems like some kind of timeout is happening and the job is stuck several minutes during releasing the lock.

What's expected?

Executed successfully

What do you get instead?

Here is an output log of our application:

🔒 Acquiring lock ... done

Checking for pending Craft and plugin migrations ...

No new migrations found. Your system is up to date.

Applying changes from your project config files ...

Finished applying changes

Checking for pending content migrations ...

No new migrations found. Your system is up-to-date.

Error: Failed to read from socket.

Redis command was: EVAL if redis.call("GET",KEYS[1])==ARGV[1] then

    return redis.call("DEL",KEYS[1])
else
    return 0
end 1 39e163799360c1e938270ea668ec756fef1cb xiRbWU3NHCxPN3ucss92

PHP Fatal error:  Uncaught yii\redis\SocketException: Failed to write to socket. 0 of 201 bytes written.

Redis command was: *5
$4
EVAL
$101
if redis.call("GET",KEYS[1])==ARGV[1] then
    return redis.call("DEL",KEYS[1])
else
    return 0
end
$1
1
$37
39e163799360c1e938270ea668ec756fef1cb
$20
xiRbWU3NHCxPN3ucss92
in /app/vendor/yiisoft/yii2-redis/src/Connection.php:827

Additional info

Q A
Yii version 2.0.45
PHP version 8.0.19
Craft Version Craft Pro 3.7.41
Operating system Linux 5.4.0-1068-azure

When does the issue occur?

Everytime when we start a seperate migrate job in a k8 azure environment deployment.

`apiVersion: batch/v1 kind: Job metadata: name: {{ include "craftcms.fullname" . }}-migrate-{{ .Values.image.tag | default .Chart.AppVersion }} labels: {{- include "craftcms.labelsJob" . | nindent 4 }} spec: template: spec: automountServiceAccountToken: false containers:

What do you see?

The Job seems to run forever. 10 min and up and shows the following stack trace:

`kubectl logs -n craftcms jobs.batch/craftcms-migrate-e1e86842eba4a3bf6d0fb0523804045e9646249f Found 2 pods, using pod/craftcms-migrate-e1e86842eba4a3bf6d0fb0523804045e9646249f-czbcq 🔒 Acquiring lock ... done

Checking for pending Craft and plugin migrations ... No new migrations found. Your system is up to date.

Applying changes from your project config files ...

Checking for pending content migrations ... No new migrations found. Your system is up-to-date.

Error: Failed to read from socket. Redis command was: EVAL if redis.call("GET",KEYS[1])==ARGV[1] then return redis.call("DEL",KEYS[1]) else return 0 end 1 39e163799360c1e938270ea668ec756fef1cb 3gUUw52QXxS0QE_4vri1 PHP Fatal error: Uncaught yii\redis\SocketException: Failed to write to socket. 0 of 201 bytes written. Redis command was: *5 $4 EVAL $101 if redis.call("GET",KEYS[1])==ARGV[1] then return redis.call("DEL",KEYS[1]) else return 0 end $1 1 $37 39e163799360c1e938270ea668ec756fef1cb $20 3gUUw52QXxS0QE_4vri1 in /app/vendor/yiisoft/yii2-redis/src/Connection.php:827 Stack trace:

0 /app/vendor/yiisoft/yii2-redis/src/Connection.php(796): yii\redis\Connection->sendRawCommand('*5\r\n$4\r\nEVAL\r\n$...', Array)

1 /app/vendor/yiisoft/yii2-redis/src/Mutex.php(146): yii\redis\Connection->executeCommand('EVAL', Array)

2 /app/vendor/yiisoft/yii2/mutex/Mutex.php(88): yii\redis\Mutex->releaseLock('craft-up')

3 /app/vendor/craftcms/cms/src/mutex/Mutex.php(75): yii\mutex\Mutex->release('craft-up')

4 /app/vendor/yiisoft/yii2/mutex/Mutex.php(88): craft\mutex\Mutex->releaseLock('craft-up')

5 /app/vendor/craftcms/cms/src/mutex/MutexTrait.php(82): yii\mutex\Mutex->release('craft-up')

6 /app/vendor/yiisoft/yii2/mutex/Mutex.php(57): craft\mutex\Mutex->release('craft-up')

7 [internal function]: yii\mutex\Mutex->yii\mutex{closure}()

8 {main}

Additional Information: Array ( )

thrown in /app/vendor/yiisoft/yii2-redis/src/Connection.php on line 827`

What was the expected result?

The Job should display a finish state and release the log: 🔓 Releasing lock ... %

yii-bot commented 2 years ago

Thanks for posting in our issue tracker. In order to properly assist you, we need additional information:

Thanks!

This is an automated comment, triggered by adding the label status:need more info.

siggelfredo commented 2 years ago

@samdark Additional Infos provided. Please review. Thanks

samdark commented 2 years ago

Well, your redis instance doesn't answer timely. No idea why. Is there anything in its logs?

siggelfredo commented 1 year ago

I talked to the DevOps from the clients Azure infrastructure team. Unfortunately there’s no dedicated Redis log file available. Therefore I can’t provide you any specific redis errors. What we have discovered so far is a relation between the./craft up and the Redis mutex lock. We changed the migration job and replaced ./craft up with


              ./craft migrate/all --interactive=0 &&
              ./craft project-config/apply --interactive=0

Afterwards we triggered the clear all caches command manually and that worked without any problem. 
It’s seems to be a craft internal problem and not a specific Redis issue.

samdark commented 1 year ago

Then please create an issue in Craft repository. I'll close this one.

Thanks for update.