Closed zsd-it closed 5 years ago
hi, 当我使用 fork 处理 10w条数据时 , 一共开启 10 个子进程. redis config
fork
'redis' => [ 'class' => 'yii\redis\Connection', 'hostname' => 'localhost', 'port' => 6379, 'database' => 0, 'retries' => 3 ],
每当子进程成功处理一条数据 ,就执行 Yii::$app->redis->incr($success_num_key); 并写入 一条 log success
Yii::$app->redis->incr($success_num_key);
success
最终结果:
redis:
127.0.0.1:6379> get batch_send_message_success_9 "100008"
log:
vim :%s/success//gn 100000 matches on 100000 lines
当我看到源码时发现:
private function sendCommandInternal($command, $params) { $written = @fwrite($this->_socket, $command); if ($written === false) { throw new SocketException("Failed to write to socket.\nRedis command was: " . $command); } if ($written !== ($len = mb_strlen($command, '8bit'))) { throw new SocketException("Failed to write to socket. $written of $len bytes written.\nRedis command was: " . $command); } return $this->parseResponse(implode(' ', $params)); } /** * @param string $command * @return mixed * @throws Exception on error */ private function parseResponse($command) { if (($line = fgets($this->_socket)) === false) { throw new SocketException("Failed to read from socket.\nRedis command was: " . $command); } $type = $line[0]; $line = mb_substr($line, 1, -2, '8bit'); switch ($type) { case '+': // Status reply if ($line === 'OK' || $line === 'PONG') { return true; ......
这里首先 调用了 fwrite 然后 fgets 再 fgets时报错 . 我猜测因为这里不是原子操作, 再调用 fwrite后 redis链接断开, 导致报错, 然后调用了下面这段代码 ,最后引起 得到得数字比想象中的要大.
fwrite
fgets
public function executeCommand($name, $params = []) { ..... if ($this->retries > 0) { $tries = $this->retries; while ($tries-- > 0) { //调用了这里 try { return $this->sendCommandInternal($command, $params); } catch (SocketException $e) { \Yii::error($e, __METHOD__); // backup retries, fail on commands that fail inside here $retries = $this->retries; $this->retries = 0; $this->close(); $this->open(); $this->retries = $retries; } } } return $this->sendCommandInternal($command, $params); }
PHP | 5.6.31 redis| Redis server v=4.0.6 yii2-redis | "yiisoft/yii2": "~2.0.14"
How can i fix it ? thanks .
if you enable retries, a single function call in yii is not an atomic operation in redis, that is expected and there is not way around that.
hi, 当我使用
fork
处理 10w条数据时 , 一共开启 10 个子进程. redis config每当子进程成功处理一条数据 ,就执行
Yii::$app->redis->incr($success_num_key);
并写入 一条 logsuccess
最终结果:
redis:
log:
当我看到源码时发现:
这里首先 调用了
fwrite
然后fgets
再fgets
时报错 . 我猜测因为这里不是原子操作, 再调用fwrite
后 redis链接断开, 导致报错, 然后调用了下面这段代码 ,最后引起 得到得数字比想象中的要大.PHP | 5.6.31 redis| Redis server v=4.0.6 yii2-redis | "yiisoft/yii2": "~2.0.14"
How can i fix it ? thanks .