Processing Separation

[jrgns]

The only real blocker for merging the code basis is that I'd like to have separate nodes for the API and Workers like we have now, to keep the API as responsive as possible. Having Worker and API processes on the same box can affect response time negatively:

• Since the worker workload peaks every hour, it will affect the response time of the API negatively.
• Depending on the workload, the worker nodes run out of CPU credits. This slows down the processing of the queues, but doesn't affect the API directly. If the nodes running the API runs out of credits, it affects the API negatively.

The system requirements for the API and workers are also different, but combining those shouldn't be an issue.

API downtime because of deploys should also not be an issue anymore. We're now running two API nodes, and the load balancer is adjusted accordingly during deployment.

@jsuchal Thoughts?

[jsuchal]

The idea here is to make one repo and two deploy different apps (api/ workers) on separate instances. API and workers. Not sure how we can managed this on AWS, but this would help us a lot.

If we move the app to ESB we can use the rolling update feature out of the box. You are problably more experienced with AWS. Is this doable?

[jrgns]

I'll have to research the solution. I'll include looking at ESB in that.

How do you do this on other projects? You said you've worked on other projects structured like this before?

[jsuchal]

@jrgns we normally use capistrano and there is no issue there.

[jrgns]

https://github.com/phstc/shoryuken/issues/169

[jrgns]

Codeship can trigger one or more deployments per push on a branch, so it looks like we can do the following everytime we push production:

1. Run tests through CodeShip
2. Deploy 1 pushes the whole app to Web instances. Let the startup scripts only start puma, and not shoryuken.
3. Deploy 2 pushes the wholl app to Worker instances. Let the startup scripts only start shoryuken, and not puma.
4. The startup scripts can use pre-set environmental variables to determine if they're on a Worker or Web instance.

We can even go so far as having more fine grained environmental variables for the workers so that you can spin up a worker that only runs a subset of the workers. We can, for instance, have workers that only process data, and workers that only process notifications.