FoundatioFx / Foundatio.Redis

Foundatio Redis
Apache License 2.0
98 stars 31 forks source link

Replace removeByPrefix Lua script with Scan #79

Closed JulienDeroff closed 10 months ago

JulienDeroff commented 1 year ago

Resolves the issue https://github.com/FoundatioFx/Foundatio.Redis/issues/78

Description of the change

Replace the Lua script that removes keys based on a prefix with multiple calls. Bulk calls made to remove the detected keys by chunks of 2500.

Note: 2500 was chosen empirically after several rounds of testing. The value can be adjusted.

SCAN is being used over KEYS to avoid blocking operations.

Motivation

According to the official documentation: https://redis.io/commands/eval/

Important: to ensure the correct execution of scripts, both in standalone and clustered deployments, all names of keys that a script accesses must be explicitly provided as input key arguments. The script should only access keys whose names are given as input arguments. Scripts should never access keys with programmatically-generated names or based on the contents of data structures stored in the database.

After some testing, the issue happens when trying to remove between 7500 and 8000 entries by prefix.

CLAassistant commented 1 year ago

CLA assistant check
All committers have signed the CLA.

niemyjski commented 10 months ago

The error message was never logged either: {"ERR Error running script (call to f_b8b6b7b2f77bc4399234cb5cbcf1f8cf083db8c5): @user_script:2: user_script:2: too many results to unpack"}

niemyjski commented 10 months ago

Thanks for the PR!