KeyDB support

[toonvd]

Hi Colin

At the moment, when using Magento, a lot of references in sets are floating around while the actual entry they refer to is already expired. These could be countered in KeyDB by using the expires on subkeys functionality. You could also segment the entries in hashes which would be a great improvement to performance since there are less loose keys to traverse.

I was thinking about forking and implementing these changes (for testing purposes at first). The end result could be kept in a separate repo but both would need maintaining. On the other hand, detection of subkey support could also be implemented and switched to.

I would love to see your thoughts on this and would be happy to contribute.

[colinmollenhour]

Hi Toon,

Very interesting, I wasn't even aware of KeyDB but it looks very impressive!

This definitely would be a great use for the EXPIREMEMBER command.

I've not personally observed that loose keys are a major issue, but I've never done any analysis either. Do you have any numbers on what percentage of keys you typically have that are orphaned?

Is KeyDB faster overall in a Magento context? Is it a complete drop-in replacement?

[toonvd]

HI Colin

I have tested and it is indeed dropin. I noticed performance increase but want to confirm with average numbers. Loadtesting results will follow this week.

When it comes to the keys, I am going to write a script and run it on a few clients. We have some clients with 24/7 traffic that don't flush the cache daily, for those, I do see a pretty large amount of orphaned keys.

[toonvd]

Hi Colin

Investigating orphaned keys gave some interesting results. Here we have a client that does not flush cache regularly:

Ok: 1687321
Orphaned: 9722884
576.23202698242

Here are some other random clients:

Ok: 45785
Orphaned: 5700
12.449492191766

Ok: 148409
Orphaned: 222
0.14958661536699

Ok: 480156
Orphaned: 131568
27.401094644241

Code used:

<?php

// redis cache uses zc:ti:

$redis = new Redis();

$redis->connect($argv[1]);

$iterator=null;

$ok=0;
$orphaned=0;

while ($iterator !== 0) {
    $keys = $redis->scan($iterator, 'zc:ti:*');
    foreach ($keys as $key) {
        $members = $redis->smembers($key);
        foreach($members as $member) {
            if($redis->exists('zc:k:' . $member)){
                $ok++;
                continue;
            }
            $orphaned++;
        }
    }
}

echo 'Ok: ' .  $ok . "\n";
echo 'Orphaned: ' .  $orphaned . "\n";

echo ($orphaned / $ok) * 100;

[colinmollenhour]

Interesting, indeed... It would seem the garbage collection is not being run at all.. I forget what mechanism triggers this or maybe there isn't one.. if not it could easily be added to a cron job of some sort.

While it looks like some nice improvements could be made I'm just not having any issues with the current state in my installations and I'm pretty happy with Redis as-is, but I'm happy to review PRs and continue discussion if you're doing the heavy lifting.

[toonvd]

I will start by forking and doing adjustments as soon as I have the needed time. For now, let's close this issue and maybe discuss this elsewhere when I have made some progress.