Concurrency issues in rate limiting and quota support

[camann9]

The rate limiting and quota data is integrated in the session (key) objects. This leads to various problems, see the following scenarios.

Scenario A, two tyk nodes:

• request arrives on node A. Session object is retrieved. Allowance is e.g. 10
• request arrives on node B. Session object is retrieved. State is the same as in node A (allowance=10)
• allowance is decremented in node A and state is saved (allowance = 9)
• allowance is decremented in node B and state is saved (allowance = 9)
• allowance has only been decremented once

Scenario B, tyk node and tky analytics:

• request arrives on node A. Session object is retrieved. Allowance is e.g. 10
• password change is requested in analytics API. Session object is retrieved (allowance=10)
• allowance is decremented in node A and state is saved (allowance = 9)
• session with changed password is saved by analytics (allowance = 10)
• allowance has not been decremented at all

These scenarios are quite likely to happen in high-load environments and apply to both quotas and rate limits (since they are essentially the same thing). A fix would be to remove the rate limit information from the session object and put it into a quickly expiring redis key (or two keys because we basically have two rates, one for short term and one for long term).

See the "Pattern" part in the redis documentation for INCR that discusses exactly this issue: http://redis.io/commands/incr

[camann9]

Thanks :-) . It's good that you reference the issue from the commit.

• Actually I think the commit doesn't solve the problem completely since it still writes back the session data to the store. So the concurrency with the dashboard issue is still unsolved.
• Writing back is probably necessary since rate limiting is still handled the old way in the session storage. Should also be changed to the new method. Then the write back could be removed, improving performance because we save one database roundtrip.
• The transaction you added in "RedisStoreManager.decrement" doesn't have any effect since the DECR command is atomic anyways. I think you can remove the transaction.
• session_manager:125 The quota is reset to the maximum quota but it should be reset to max-1 since one call is already used.

[lonelycode]

I actually realised in the shower this morning that this commit doesn't fully solve the issue anyway, since we set the value of the Quota if the key is not found, strictly speaking, Scenario A still applies since a concurrent set of requests could set the same key to a maximum, now this isn't too much of an issue, but theoretically speaking you could bypass quotas with some clever timing.

I'm considering switching it around to be an INCR instead of a DECR, then there's no SET required (then just subtract Max from Remaining to update the session cache, however the race condition that is mentioned in the pattern applies where there is a risk of a floating key, though I feel it's unlikely to occur.

I think Scenario B is solved since the rate limiter uses the Redis store as the "truth" and the session store as a record, so if a password reset resets the quota counter, if the key is already present then it will just DECR (or INCR, as the case will be), the actual value in the session is irrelelvant except for in the reply and health-checks, which can't be guaranteed values.

Will send some more time on this today.

[lonelycode]

Tests passing, quota and rate limiter now redis based with finer-grained quota reset commands in the API. :-)

[camann9]

Awesome. Just a few things:

• The method session_manager.IsQuotaExceeded should be unused now, could be deleted
• In the current version of the quota algorithm the quota is handled like the rate limit. This means that QuotaRenews is ignored. The field could be removed. This also has the consequence, that a user could not set it manually any more (e.g. if he wants the renew to always occur at midnight).
• RedisStorageManager.Decrement could be removed now, no?
• RedisStorageManager.IncrememntWithExpire calls RedisStorageManager.Decrement if it has to reconnect (line 267). I assume this is a typo.
• Can't you just remove the reconnect attempts in RedisStorageManager? Does r.pool.Get() not deliver a connection reliably?

[lonelycode]

1. Yes
2. QuotaRenews is used to report back to health check headers, it is also used to set the initial rate limit. User can set it manually, updating a session via API will reset the quota in Redis (unless new param ?suppress_reset=1 is sent with request), this basically means quota renewals can be scripted to occur whenever if needed). This was actually how it all behaved, so we're preserving exiting functionality, but adding a workaround for session updates that shouldn't affect quota.
3. Yes it can, though not a priority
4. Eek! Fixed in master.
5. Leaving it as is for now

[camann9]

On 2.: QuotaRenews is not set any more in IsRedisQuotaExceeded. Thus it is not updated any more anywhere (since IsQuotaExceeded is not used any more). Or am I overlooking anything?

[lonelycode]

You're right, it isn't set, and really should be, it's used in reporting current session state back to the user through the rate-limit endpoint and reply headers:

https://github.com/lonelycode/tyk/blob/b28c02a59e9deec2bba398b1b3bdc2b4c50b31cc/api.go#L1204

https://github.com/lonelycode/tyk/blob/77f4f7d5e23d78a46f4d02a07044c6402fc9163a/tyk_reverse_proxy_clone.go#L185

Have just pushed an update to master branch.

[camann9]

Yeah, that solves it :-)