StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.84k stars 1.5k forks source link

How to resume enumerate key from last breakpoint? #2717

Closed maomaomqiu closed 1 month ago

maomaomqiu commented 1 month ago

Hi, I am working on a task to purge persistent keys generated due to legacy bugs, below is main logic

...
(1) var keys = server.KeysAsync(pattern: pattern).GetAsyncEnumerator();

(2) batchExpireBuffer.Add(keys.Current.ToString()); 

(3) every 50 keys, BatchSetExpire(batchExpireBuffer)

Consider the total amount of keys need purge is huge (could up to 50 million), We start enumerate from key0, suppose the collection for enumerator is {key0, key1, key2, ...}, once there are exception happened in (1) - (3), and before exception happened, we already finished key1, is there any way to get some information, so that when run next job, we can start from key2?

maomaomqiu commented 1 month ago

I have thought about a way, log cursor value and pageoffset when exception happened

maomaomqiu commented 1 month ago
var currentKey = keys.Current.ToString();
IScanningCursor cursor = (IScanningCursor)keys;
long cursorvalue = cursor.Cursor;
long pageoffset = cursor.PageOffset;