Automatic deployment of branches

[KimStebel]

With these changes, new branches are automatically deployed to docs-testb.cloudant.com/$branch. Also fixes some quoting bugs in the deployment scripts.

[wohali]

Hey @evansde77 @mattwhite and @rnewson what are your thoughts here? Dashboard also deploys the same way right now, and I'm somewhat uncomfortable with the approach. Especially in the Dashboard, what lives at a deploy- branch looks very different than what's on master.

When I originally designed the Cloudant-backed artifact repository system the idea was to have a proper place where built assets lived. That could be binaries or built docs or whatever. It seems we've fractured into 3-4 different ways of doing things: this branch-based approach, pip and a pypi server for Pythonny things, we're still on tarballs for dbcore / similar, packages for binaries....and maybe that's OK. I am just curious if we should be trying harder to normalise our approach for built assets that don't turn into binaries like the dashboard, documentation and so on.

@KimStebel sorry to hijack your PR here but I think it bears discussion, esp. in light of the branch-deployment-to-production approach.

[evansde77]

@wohali My approach is to build in the git flow automated release build candidate with uploads to our pypi server and then eventually auto deploy to a SAPI staging cluster for integration tests. Basically our master branches end up being very clean release artifact repositories. I guess with the dashboard artifact there must be a pypi equivalent right? Or even just a web server with a directory based versioning system? Or just use git flow and let github serve the release candiates from master? In terms of deploying versions, I want the version controlled by the chef environment, eg in wilson now, I can just flip the version field in the env and it will deploy the new artifact everywhere using pip install from the pypi server

[mattwhite]

We could actually host all packages and assets on our apt server (our apt server is actually s3, but that's another topic). We currently use it to host third-party .debs, ruby gems (that have been converted to .debs), and python packages (e.g. python-glee). We could use it for dbcore and everything else living in deployer2.cloudant.com:/opt/repo/builds. Dashboard assets could also be turned into .debs. This would give us a single and consistent way of deploying things.

I'm also ok with separate apt, pypi, and gem servers.

[wohali]

I like @mattwhite's idea here.

So this really comes down to Docs and Dashboard agreeing as they are the major areas that differ, though a few lingering dbcore things need revising (/cc @rnewson).

@garrensmith @kimstebel what are your feelings about deploying NOT through a build-then-push-to-deploy-branch model? Instead it looks like a (Jenkins-automated) build-to-zip/tar, then promote-and-upload model instead. I know it's an extra step but it brings our teams in line with the other teams.

One other point - this is a public PR - please keep that in mind.

[garrensmith]

I'm ok with that, in fact this would probably be a lot better.

On Sat, Oct 18, 2014 at 11:22 PM, Joan Touzet notifications@github.com wrote:

I like @mattwhite https://github.com/mattwhite's idea here.

So this really comes down to Docs and Dashboard agreeing as they are the major areas that differ, though a few lingering dbcore things need revising (/cc @rnewson https://github.com/rnewson).

@garrensmith https://github.com/garrensmith @kimstebel https://github.com/kimstebel what are your feelings about deploying NOT through a build-then-push-to-deploy-branch model? Instead it looks like a (Jenkins-automated) build-to-zip/tar, then promote-and-upload model instead. I know it's an extra step but it brings our teams in line with the other teams.

— Reply to this email directly or view it on GitHub https://github.com/cloudant-labs/slate/pull/5#issuecomment-59630051.

[KimStebel]

Let me first clarify what is actually happening here. Nothing gets deployed into production from any of the branches. All this PR does is extend the automatic deployment of the master branch (which is already in place) to all branches of the repository. The branches are deployed into a staging environment at docs-testb.cloudant.com/$branch, which no customer will ever see. I think we should automate as much of the deploy and build process as possible, so I don't see why we would take this step backwards. Also, I don't see a good argument why different projects with different requirements should all follow the same build/deploy process. @bradnoble, what do you think?

[bradnoble]

I like the idea of bringing the processes together, as many of us work on multiple projects and having one way to do it keeps any of us from having to ask, "how does this work?" or the even worse outcome, which is to assume and be wrong. That said, I do think this is a states rights issue, and each project should be able to decide for themselves whether or not to borrow the process from the mothership or go their own way for their own valid reasons.

[mattwhite]

I think the issue here is that after being deployed, services need to be managed by the ops team, and having a nonstandard setup can increase MTTR if the service goes down for some reason. A recent example of this is wilson moving from supervisord to runit because it makes the service easier for ops to manage.

[KimStebel]

@mattwhite: that's not an issue, since we are merely creating databases on an MT account.

[KimStebel]

Had a chat with Joan and cleaned up some misunderstandings as well as agreed to moving the build process to Jenkins and doing deployments via Chef in the future. For now, this gets merged.