sentry-extensions / cyclops

Cyclops is meant to be a high performance barrier in front of Sentry (http://getsentry.com).
http://sentry-extensions.github.io/cyclops/
163 stars 15 forks source link

Inefficiency in RedisStorage.get_next_message() #35

Open mkrysiak opened 6 years ago

mkrysiak commented 6 years ago

https://github.com/sentry-extensions/cyclops/blob/9136a4f8121999e3d1d7996690a8516b150f7e01/cyclops/storage.py#L95-L103

get_next_message() uses redis.smembers() to get a list of all project_id's, and project_ids are added to the set in put(), but project_id's are never removed from the set if there are no request in the project_id's queue. So get_next_message() may be calling redis.rpop() on a key that no longer exists.

Another option may be to use redis.scan() for keys that match "cyclops:queue:*", and randomly select from that list for the next message. This would guarantee that redis.rpop() is called on a queue that contains at least one message.