Imgur / incus

incus is a middleware for distributing messages via websockets, long polling, and push notifications
MIT License
361 stars 37 forks source link

Impl. redis active gophers #52

Closed jacobgreenleaf closed 9 years ago

jacobgreenleaf commented 9 years ago

This PR re-adds support for the Presence commands.

We need to send redis commands (ZREM) every time a websocket client disconnects, to mark them as inactive for that client.

Before, we were naively just using the redis connection pooling library, but there are way too many websocket events to do that; there are roughly ~200 websocket disconnects per second at peak, which would mean that every single disconnect would connect to redis, issue ZREM, and then disconnect. Lots of connections would time out and it was just bad.

This PR makes it so that there is a fixed-at-runtime number of gophers (redis consumers) that issue commands on behalf of websockets to Redis for those presence commands. That number is 8 currently, but is configurable (redis_activity_consumers). Websockets forward commands (RedisCommands) to a queue process (RedisConsumerPendingList). That pending list is a linked list of pending redis commands. The queue process dispatches them to a number of workers that each perform the command and report back.

One concern is that the rate of websocket disconnections (about 200/sec) could be faster than the consumers can dispatch to Redis; if it takes more than 40ms on average from the moment the websocket disconnects to the moment the result returns, then the redis command queue will grow without bound. I have added a StatsD metric for the queue length (incus.pendingactivityredislen), and it would be prudent to add another metric for how long each command takes from start to finish.

Also, nothing stops us from using these for any arbitrary redis commands that websockets do.

briankassouf commented 9 years ago

:+1:

lospro7 commented 9 years ago

:+1: