search

dblock / slack-gamebot

Ping pong, chess, pool and other games bot for Slack.
MIT License
129 stars 103 forks source link

Horizontal scalability? #81

Closed timurkhafizov closed 1 year ago

timurkhafizov commented 8 years ago

As far as I see web sockets run in a single thread within web dyno. What if i need to increase dyno size? Will app open dup connections?

dblock commented 8 years ago

It would, and it would be very bad.

In fact Heroku recycles dynos and I believe we do see very brief moments where you're having two instances running.

I've been thinking about how to deal with this. For Heroku I think the easiest would be to declare the number of working dynos in ENV and distribute load that way. The next step is probably to have a more transactional way of "starting" a service and having dynos compete for locks on start, then try to rebalance at runtime automagically. That problem alone could be a great Ruby gem.

timurkhafizov commented 8 years ago

Basically we have https://github.com/schneems/the_lone_dyno which can help us to isolate thread in a single dyno while puma's workers can run in a multiple.

Did you ever count how much websocket connections server can handle in a standard Heroku dyno?

dblock commented 8 years ago

I think isolating workers to 1 dyno won't do much for slack-gamebot unless someone starts heavily using the API.

I think we should be able to do many hundreds at the very least. Right now I have an instance on http://playplay.io that isn't breaking a sweat with stable RAM at about 300MB (bot classes are singletons) with 100+ teams.

dblock commented 8 years ago

The constructs we need from each dynos perspective is:

  1. Tell me what the optimal number of connections should I be starting based on the number of workers alive.
  2. If I am above that number, I will stop some.
  3. If I am below that number, give me the next un-started connection to start.

The system should be stable, ie. the optimal number should be a little bigger than total connections / number of workers so that connections don't constantly get shutdown and restarted elsewhere.

timurkhafizov commented 8 years ago

Yeah, I was thinking the same. Anyway we should store connection ids somewhere then. It maybe Redis and we can just store bot token there or something similar.

The only thing I aware of is that we can't see if dyno crashed and did not update that info in Redis.

timurkhafizov commented 8 years ago

Probably it can be any another NoSQL DB providing persistence.

dblock commented 8 years ago

Slack-gamebot already has a MongoDB store, you can use atomic updates and https://github.com/afeld/mongoid-locker. Would love some PRs!

dblock commented 8 years ago

I've introduced https://github.com/dblock/slack-gamebot/blob/master/Procfile.heroku which can be used to split the web and the worker process, so now web ones can be scaled horizontally. You still don't want multiple workers.