search

codecov / engineering-team

This is a general repo to use with GH Projects
1 stars 1 forks source link

Create CRON job to delete Rate Limit GQL Keys #2588

Open ajay-sentry opened 1 week ago

ajay-sentry commented 1 week ago

After the GQL Rate Limit initiative was rolled out, we began seeing some instances where users were being locked out of their accounts indefinitely. From our investigation, we realized that for some reason sometimes redis keys pertaining to a user just don't expire. This could be due to a variety of reasons, but in any-case we want an easy way to automatically restore these user's access.

The purpose of this ticket is to create a CRON job that will automatically delete redis keys for all users and ip's hourly, so a user will at most be locked out for an hour at a time.

There are two prefixes that we will need to search for keys for at the moment:

The former is used for users who are logged in, and the latter for anonymous users.

trent-codecov commented 6 days ago

@ajay-sentry While we can do this, can we not put a TTL on these redis keys? Typically you accomplish this by just setting a 1 hour TTL when you write the key. That way after an hour, redis will take care of the deletion.

ajay-sentry commented 6 days ago

@trent-codecov Great question, sorry I should have elaborated further that we do currently have a ttl for the Redis keys of 1 minute. For the most part this works as designed, but there are maybe 2-3 individuals per week who for some reason have a key that just never expires

Relevant code snippet: https://github.com/codecov/codecov-api/blob/4df7869ce55aeaf4330f086b23aa91ff9d1c2b0a/graphql_api/views.py#L332