revalidateTag is very slow when there are lots of keys

[zipme]

Brief Description of the Bug When using redis-string, and once redis has lots of keys (we have 18745) the revalidateTag could take 10 seconds to run

Severity [Major]

Frequency of Occurrence [Always]

Steps to Reproduce Provide detailed steps to reproduce the behavior, including any specific conditions or configurations where the bug occurs:

1. Populate redis with more than 10000 keys
2. Call revlidateTag

Expected vs. Actual Behavior The revalidation should hopefully take less than a second but it takes way longer (sometime 10s+)

Environment:

• OS: [Linux]
• Node.js version: [e.g., v20.11.0]
• @neshca/cache-handler version: [e.g., 1.3.2]
• next version: [e.g., 14.2.4]

[better-salmon]

@zipme, hello! Which Next.js router do you use: App or Pages?

[zipme]

Hi @better-salmon we use app router. When the cache is not primed the revalidation goes fast.

[better-salmon]

Thanks for bringing this issue to our attention. I understand the frustration caused by the long execution time of the revalidateTag process, especially with a significant number of keys in Redis.

The current revalidation process has algorithmic flaws. It uses an O(n²) algorithm to find all cache entries needing revalidation. This issue is inherited from Next.js, which recommends using this approach to delete cache entries. The implementation must use Array.prototype.includes inside a loop, which results in the observed performance issues. Here's the code from the Next.js docs:

revalidateTag(tag) {
    // Iterate over all entries in the cache
    for (let [key, value] of cache) {
        // If the value's tags include the specified tag, delete this entry
        if (value.tags.includes(tag)) {
            cache.delete(key)
        }
    }
}

Unfortunately, I don't see a way to improve this due to how Next.js uses tags.

I've made some optimizations in the latest pull request to address Redis's performance concerns. A new option, revalidateTagQuerySize, has been added to the redis-stack and redis-strings Handlers. This option allows you to specify the number of tag lists retrieved in a single query from Redis during the scanning or searching. Additionally, the default query size for hScan in redis-strings and ft.search in redis-stack has been increased from 25 to 100. This adjustment reduces the number of network roundtrips by optimizing the number of commands sent to Redis, though it increases the TCP packet size. By increasing the query size, we aim to balance command count and network roundtrips, improving performance in scenarios with many keys. In addition, I've made the updating of the sharedTags map command isolated, and it must not block other Redis commands.

Finding the best revalidateTagQuerySize for your setup can vary based on your environment and workload. Start with a value of 100 and adjust it while monitoring the performance impact. Experiment with different values to fine-tune the balance between the number of commands sent to Redis and the size of the TCP packets.

Please update to the version 1.4.0 or newer with these optimizations and test if the performance improves in your setup. Your feedback will be valuable in further refining the solution.

Thanks for your patience and understanding.

[zipme]

@better-salmon how about creating a tags manifest? Something like this maybe?

[better-salmon]

I used to utilize the tags manifest in versions before 1.0.0, but I need time to recall why I stopped using it.