search

davidbanham / field-marshal

A landlubbing rear-admiral
BSD 3-Clause "New" or "Revised" License
27 stars 3 forks source link

Improved push / launch workflow for new versions #8

Open JedWatson opened 10 years ago

JedWatson commented 10 years ago

@bladey and I deployed an app to FM today, and the processes magically started on the cavalry, which almost worked as expected ;)

The whole thing feels fairly "hands off" though, especially compared to Heroku's validation process where it tests the deployment before spinning up the new dynos.

I was thinking it would be great to show (in Quartermaster) the currently running commits, and any pushed commits after or before it. You could then explicitly tell FM to spin up a different commit on the cavalry, which it would do, then it could switch over the routing to the new processes and end the old ones after a timeout.

This would be a neat way to downgrade to a prior version if ever necessary; not sure how much that would be useful in the real world but it's always nice to have the option.

Taking the idea further; it would be neat to log when versions were up - start / end dates, duration / etc, so there's a history of what's been running when;

And we could also (related to #6) have the ability to spin up a new version on a single Cavalry slave (without routing traffic to it) in a 'test' mode; deploy it, log the output, make sure it comes up safely before actually deploying to the cavalry.

davidbanham commented 10 years ago

This relates tightly to the "modify/create manifests from quartermaster" feature, so we should build it in alongside that. Most of the functionality will be provided by that, but the extra historical data would be very handy. Good thought.

I like the idea of making staging easier, but I think that might be a documentation thing rather than a code thing. The idea of adding a second "class" of thing to deploy seems overly complex. There are two good ways to deal with staging things.

  1. Have a totally separate cluster+FM. This is probably the neatest way, in that it keeps things nicely compartmentalised.
  2. Have a separate manifests on the same cluster for prod and staging. You would have one manifest named "mygreatapp" with the routing domain "mygreatapp.com" and one manifest named "mygreatapp-staging" with the routing domain "staging.mygreatapp.com".
JedWatson commented 10 years ago

re: the staging thing, this was more about ensuring (just before launching) that it all builds successfully - I've seen a properly staged version then fail to deploy for unknown reasons (which may be as simple as npm issues). So it's more about confirming the build launches before trying to deploy it on all the cavalry.

The other thing (this is really Keystone related) is that if you're using update scripts, you want them to run (in production) on a single node and complete successfully before rolling out to multiple processes. This is arguably a problem that could / should be solved elsewhere, but having the ability to spawn a single process and watch its output before rolling out the production cluster would be a neat workaround.