yiisoft / yii2-redis

Yii 2 Redis extension.
http://www.yiiframework.com
BSD 3-Clause "New" or "Revised" License
452 stars 183 forks source link

redis inc Multiple processes error #171

Closed zsd-it closed 5 years ago

zsd-it commented 5 years ago

hi, 当我使用 fork 处理 10w条数据时 , 一共开启 10 个子进程. redis config

 'redis' => [
            'class' => 'yii\redis\Connection',
            'hostname' => 'localhost',
            'port' => 6379,
            'database' => 0,
            'retries' => 3
        ],

每当子进程成功处理一条数据 ,就执行 Yii::$app->redis->incr($success_num_key); 并写入 一条 log success

最终结果:

redis:

127.0.0.1:6379> get batch_send_message_success_9
"100008"

log:

vim  :%s/success//gn

100000 matches on 100000 lines

当我看到源码时发现:

 private function sendCommandInternal($command, $params)
    {
        $written = @fwrite($this->_socket, $command);
        if ($written === false) {
            throw new SocketException("Failed to write to socket.\nRedis command was: " . $command);
        }
        if ($written !== ($len = mb_strlen($command, '8bit'))) {
            throw new SocketException("Failed to write to socket. $written of $len bytes written.\nRedis command was: " . $command);
        }
        return $this->parseResponse(implode(' ', $params));
    }

 /**
     * @param string $command
     * @return mixed
     * @throws Exception on error
     */
    private function parseResponse($command)
    {
        if (($line = fgets($this->_socket)) === false) {
            throw new SocketException("Failed to read from socket.\nRedis command was: " . $command);
        }
        $type = $line[0];
        $line = mb_substr($line, 1, -2, '8bit');
        switch ($type) {
            case '+': // Status reply
                if ($line === 'OK' || $line === 'PONG') {
                    return true;

......

这里首先 调用了 fwrite 然后 fgetsfgets时报错 . 我猜测因为这里不是原子操作, 再调用 fwrite后 redis链接断开, 导致报错, 然后调用了下面这段代码 ,最后引起 得到得数字比想象中的要大.

 public function executeCommand($name, $params = [])
    {
      .....
        if ($this->retries > 0) {
            $tries = $this->retries;
            while ($tries-- > 0) {   //调用了这里
                try {
                    return $this->sendCommandInternal($command, $params);
                } catch (SocketException $e) {
                    \Yii::error($e, __METHOD__);
                    // backup retries, fail on commands that fail inside here
                    $retries = $this->retries;
                    $this->retries = 0;
                    $this->close();
                    $this->open();
                    $this->retries = $retries;
                }
            }
        }
        return $this->sendCommandInternal($command, $params);
    }

PHP | 5.6.31 redis| Redis server v=4.0.6 yii2-redis | "yiisoft/yii2": "~2.0.14"

How can i fix it ? thanks .

cebe commented 5 years ago

if you enable retries, a single function call in yii is not an atomic operation in redis, that is expected and there is not way around that.