cache.delete_pattern() is very slow

[selected-pixel-jameson]

When I use the cache.delete_pattern() in my application it is blocking the response and is running extremely slow. Is there a way to execute this asynchronously? Or in a more efficient manner of any kind?

[sebleier]

What version are you using? Would it be possible to throw that call in a celery task or spin off another thread?

[selected-pixel-jameson]

I can look into that. I'm running version 1.8.1

[sebleier]

At version 1.8.1, the library is using scan_iter to find all the keys that match the pattern. That's the preferred method to find keys matching a pattern. The only other way I can think of to do this faster is write a lua script, which may actually be a better pattern anyways, since it'll be atomic.

[selected-pixel-jameson]

To further expand on this it only happens when connecting to a Redis instance setup using AWS ElasticCache service. If I use it locally it works fine. I'm guessing it could also just be because that is a live instance and has much more data. I haven't tried to set this up using Celery yet.

[garvit4512]

Facing the exact same issue. However, what troubles me is that there are only a few dozen keys in my delete_pattern and still it takes atleast 5 secs to resolve. If I already know the keys, would cache.delete() be faster?

[selected-pixel-jameson]

I was not able to use the pattern delete at all because of latency issues. I ended up just deleting the cache records one by one.

[josemlp91]

The delete_pattern internally used scan_iter and by default it set the itersize equal to 10. I coud recoment raise this value, for example:

cache.delete_pattern("my_key", itersize=10000)

[jefer94]

This at least join the deletions, changing a few lines this should be fine

class CustomRedisClient(DefaultClient):

    def delete_pattern(
        self,
        pattern: str | list[str],
        version: Optional[int] = None,
        prefix: Optional[str] = None,
        client: Optional[Redis] = None,
        itersize: Optional[int] = None,
    ) -> int:
        """
        Remove all keys matching pattern.
        """

        if isinstance(pattern, str):
            return super().delete_pattern(pattern,
                                          version=version,
                                          prefix=prefix,
                                          client=client,
                                          itersize=itersize)

        if client is None:
            client = self.get_client(write=True)

        patterns = [self.make_pattern(x, version=version, prefix=prefix) for x in pattern]

        try:
            count = 0
            pipeline = client.pipeline()

            for key in itertools.chain(*[client.scan_iter(match=x, count=itersize) for x in patterns]):
                pipeline.delete(key)
                count += 1
            pipeline.execute()

            return count
        except redis_client_exceptions as e:
            raise ConnectionInterrupted(connection=client) from e

[jefer94]

Hey, I'm not known if I got it, but delete_pattern can't receive many patterns in anything like an OR, right?

[jefer94]

This solution didn't work, if the performance is a priority you must set a key with all the keys you need to delete

• delete_pattern take 0.5s
• using get_many with a collection of keys and delete_many to delete them take 0.38s the first time and 0.08s the second time, this delta should because by a conditional that I have that just is executed on the first time