StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.9k stars 1.51k forks source link

Resumable scan through keys #2425

Open adamijak opened 1 year ago

adamijak commented 1 year ago

Hi, I am trying to process keys on server in this batch pattern.

var cursor = 0;
do
{
    var keyBatch = await GetKeysAsync(cursor);
    foreach (var key in keyBatch)
    {
        ProcessKey(key);
    }
    cursor = keyBatch.NextCursor;
} while (cursor != 0);

How can I achieve this behaviour using IServer and IScanningCursor? This does not work.

var cursor = 0L;
do
{
    var counter = 0;
    var keys = server.KeysAsync(1, "user:*", 1000, cursor);
    await foreach (var key in keys)
    {
        counter++;
    }
    Console.WriteLine($"{cursor}, {counter}");

    cursor = ((IScanningCursor) keys).Cursor;
} while (cursor != 0);
mgravell commented 1 year ago

The handling of IScanningCursor here is not meant to act as discreet fetch operations i.e. individual calls to SCAN; we always hide those details. Instead, the intent is that if you need to interrupt the scan, you can capture the current state via IScannignCursor, and use those numbers as the optional arguments to a later call to Keys[Async], and you should resume from where you were. Does that make sense?

I guess if you really really want that, you could use Execute[Async]?

adamijak commented 1 year ago

I do use ExecuteAsync now, since it is only solution I could come out with. But I would really prefer If there was Scan[Async] command predefined. For the sake of consistency with Redis commands. Keys[Async] could then use Scan[Async] command under the hood and return IAsyncEnumerable

adamijak commented 1 year ago

Btw the reason why I do this, is because I pass the cursor between different server-less functions. So basically I do not have control over the topmost while loop.