yiisoft / yii2-redis

Yii 2 Redis extension.
http://www.yiiframework.com
BSD 3-Clause "New" or "Revised" License
451 stars 184 forks source link

读取mysql数据存储入redis发生php内存溢出 #125

Closed Y2k38 closed 5 years ago

Y2k38 commented 7 years ago

\Yii::$app->cache->set($key, $value)操作redis存储数据发生异常,内存占用不断涨高,最终超出系统限制的128M; mysql数据量:> 1,000,000,每次读取1000条数据 \Yii::$app->cache->set()使用次数:每条记录存储共6次 其他: 运行方式为command模式; 屏蔽redis存储操作,PHP内存占用约19M; 使用PHP扩展phpredis,PHP内存占用亦约为19M

Additional info

Q A
Yii vesion 2.0.9
PHP version PHP7.0.3
Operating system 阿里云VPC网络服务器/CentOS6.9
cebe commented 7 years ago

google translate:

\ Yii :: $ app-> cache-> set ($ key, $ value) operation redis storage data anomalies, memory consumption continues to rise, and ultimately beyond the system limit of 128M; mysql data volume:> 1,000,000, each reading 1000 data \ Yii :: $ app-> cache-> set () Used: 6 times per record other: The mode of operation is command mode; Mask redis storage operation, PHP memory occupies about 19M; Using PHP to extend phpredis, PHP memory is also available for about 19M

Could you provide some code you are using? If memory increases in a loop, check if you keep some references somewhere that are unnecessary and try to unset them.

yii-bot commented 7 years ago

Thanks for posting in our issue tracker. In order to properly assist you, we need additional information:

Thanks!

This is an automated comment, triggered by adding the label status:need more info.

Y2k38 commented 7 years ago

main function:

foreach ($tables as $table) {
    $total = 0;
    $table = 'user_' . $table;

    echo "caching table $table ";

    for ($i = 0; ; $i++) {
        $qry = sprintf("select * from %s limit %d offset %d", $table, 1000, 1000 * $i);
        $data = $model->getDb()->createCommand($qry)->queryAll();

        if (empty($data)) {
            break;
        }

        foreach ($data as $k => $v) {
            $total ++;
            UserCache::cache($v, false);
        }

        unset($data);

        echo ".";
    }

    echo " total -> $total\n";
}

UserCache::cache():

public static function cache($user, $detail = false)
{
    if (is_object($user) && $user instanceof User) {
        $user = $user->toArray();
    }

    if (! empty($user)) {
        $user['username'] = strtolower($user['username']);
        $key = \Yii::t('app', 'Username Cache Key', $user['username']);
        \Yii::$app->cache->set($key, $user['id']);

        $key = \Yii::t('app', 'User Id Cache Key', $user['id']);
        \Yii::$app->cache->set($key, $user['username']);

        ...
    }
}

like this, when i comment out the code "\Yii::$app->cache->set($key, $value)", it works normally. i can't find out what's wrong, but it works in my VM mathine

cebe commented 6 years ago

if you have logging enabled, Yii::$app->cache->set will log a trace level message, how did you configure logging?

yii-bot commented 5 years ago

It has been 2 or more weeks with no response on our request for more information. In order for our issue tracker to be effective, we are closing this issue.

If you want it to be reopened again, feel free to supply us with the requested information.

Thanks!

This is an automated comment, triggered by adding the label expired.