holman
commented
8 years ago Use deploy locking to lock deploys, roll the code out to 10% of 1 customer, then roll the code out to 100% of 1 customer, then roll the code out to all customers. This could add inertia to deploys though.
I don't have a whole lot of experience with a very segregated production like this, but this seems to me at first blush to be a decent approach. If you look at production as being less about "20 customers" and more about "20 servers in production", the whole structure reads like a more traditional deploy model. You could do incremental rollout on a single customer first (maybe the one who pays you the least 😃) and then roll to everyone overall.
The other option if the percentage rollout is really bogging you down is to again, just treat everything as one gigantic version of production and deploy to everything all at once. Your deploy methodology should be one-click or one-command and automated, so it shouldn't necessarily be more convoluted to deploy to all subsets of production at once (and rollback should be the same).
First of all, I really enjoyed your Deploying Software article!
We are a what I guess can be described as a data-processing-as-service within the Airline industry. Due to the sensitive nature of the data each of our customers has a identical, completely separate infrastructure (this is a hard requirement from their side). This means that with 20 customers we run 20 copies of the same infrastructure.
We automate heavily since any manual labor would quickly drown us. But we haven’t yet found the simplest way to deploy. Right now we end up with running multiple versions of the same API with some customers lagging behind. This also mean that the customers may be updates in different fashion, i.e:
Customer A: Version 1.0 -> 1.1 -> 1.2 -> 1.3 -> 1.4 Customer B: Version 1.0 -> 1.2 -> 1.4
And us keeping track over the versions per customer in a spreadsheet.
Clearly not an ideal solution.
Some options:
Use auto-deploy on push to master on all customers, but rollback is harder if you need to do it 20 times (automatically or not). It’s also slightly harder to monitor. Use deploy locking to lock deploys, roll the code out to 10% of 1 customer, then roll the code out to 100% of 1 customer, then roll the code out to all customers. This could add inertia to deploys though. Also, you need to be able to select which customer to deploy to etc.
We’ll most likely end up extending our deploy interface (which communicates with Travis, Github etc) but I want to make sure we aren’t missing a way to make it even simpler.
Any ideas?
Would love to use you as a consultant for a day to improve our deploy process, but we're bootstrapped so it will have to wait a while.