manage dev, prod, vm concerns with git branches

[btbonval]

Instead of having files scattered about that are slightly different but need to be updated by hand when there are common changes, why not make use of git branching?

master branch can contain prod files. dev branch can contain dev files. vm branch can contain vm fies.

If it is done correctly, such that vm merges into dev merges into master, and so long as certain files are newer on master (e.g. manage.py), then these branches can coexist peacefully and share with each other very nicely.

It'll clean up a lot of the nesting directories and reflect the dev/prod branch changes commonly seen in web services workflows (at least the ones I've seen in previous workplaces)

[AndrewMagliozzi]

Yes! Bryan, can you make a very short doc that diagrams our git workflow? It'll be good for the newbies on Thursday if possible.

On Jan 15, 2014, at 12:07 AM, Bryan Bonvallet notifications@github.com wrote:

Instead of having files scattered about that are slightly different but need to be updated by hand when there are common changes, why not make use of git branching?

master branch can contain prod files. dev branch can contain dev files. vm branch can contain vm fies.

If it is done correctly, such that vm merges into dev merges into master, and so long as certain files are newer on master (e.g. manage.py), then these branches can coexist peacefully and share with each other very nicely.

It'll clean up a lot of the nesting directories and reflect the dev/prod branch changes commonly seen in web services workflows (at least the ones I've seen in previous workplaces)

— Reply to this email directly or view it on GitHub.

[charlesconnell]

When we make a change, what branch should we make it on? Then what other branches do we have to merge that into?

[btbonval]

It turns out there was a really good article my friend shared via Google+ which explained this sort of practice really nicely with images. There were some little things in the practice I would (and have) change(d), but it was close enough. Let me find that article and post the link here.

I could write up this process myself, but lately my ability to generate coherent sentences has been faltering.

[btbonval]

The beginning is a git primer. Skip down to "the main branches" which describes how to use dev and master. Feature branches is a good section; I've been doing this with almost every feature I work on. ("almost" because sometimes I squash non-pushed branches onto master, so you might not always see I had a feature branch). http://nvie.com/posts/a-successful-git-branching-model/

At a previous workplace, we had several clients. I developed a similar git workflow, except that each client had its own "master branch". We only had one dev branch, which was merged into the client master branches as we needed to upgrade the client production code. Master branches were only forked for hot fixes, while the dev branch was forked for every feature.

We would tag master branches with the "release date": when the production system started running the code. If something went wrong, we would find the previous most recent tag and check that out, or we could rollback to any production bump by date. That's a little different than how tagging is recommended in the above link.

[btbonval]

@AndrewMagliozzi did you want me to document our current workflow or the proposed one? The proposed one is pretty nicely documented in the link above.

[charlesconnell]

That system is definitely useful for managing different versions of your code across time, as they describe with feature branches, release branches, etc. But that's different from managing different configurations for dev, vm, and prod. We don't want to merge our development settings currently in settings/dev.py into our production settings currently in settings/prod.py when we're ready for a release, we want those to always be different. I also suspect this release workflow is overkill for our comparatively simple project, if we were to use it for that purpose. Pull requests and feature branches are working well for us as it is.

[sethwoodworth]

I think that this will cause difficulty. Managing git branches and interactively rebasing a changeset is not trivial, especially for new developers. This will also remove the ability to compare dev and production versions of configuration files without checkout out several versions of the repository.

Also, if you want to add a file to the dev branch, but not to the production branch, you would have to have the production changeset describe removing that file.

This is advanced git-fu.

On Wed, Jan 15, 2014 at 3:31 PM, Charles Connell notifications@github.comwrote:

That system is definitely useful for managing different versions of your code across time, as they describe with feature branches, release branches, etc. But that's different from managing different configurations for dev, vm, and prod. We don't want to merge our development settings currently in settings/dev.py into our production settings currently in settings/prod.py when we're ready for a release, we want those to always be different. I also suspect this release workflow is overkill for our comparatively simple project, if we were to use it for that purpose. Pull requests and feature branches are working well for us as it is.

— Reply to this email directly or view it on GitHubhttps://github.com/FinalsClub/karmaworld/issues/279#issuecomment-32411778 .

[btbonval]

@charlesconnell most recent common ancestor. When you merge one branch into another, it doesn't mean all changes are overridden. So long as master's copy of that requirements text file is "newer" in the hierarchy, you can merge dev into master without those changes getting pushed.

@sethwoodworth Not sure what you mean. It's very simply merging one branch into another. Features branch off of dev, merge back into dev. When it's time for a release, dev merges into master.

[sethwoodworth]

Are you suggesting having a dev branch and a prod branch? Or am I misunderstanding

On Wed, Jan 15, 2014 at 3:41 PM, Bryan Bonvallet notifications@github.comwrote:

@charlesconnell https://github.com/charlesconnell most recent common ancestor. When you merge one branch into another, it doesn't mean all changes are overridden. So long as master's copy of that requirements text file is newer in the hierarchy, you can merge dev into master without those changes getting pushed.

@sethwoodworth https://github.com/sethwoodworth Not sure what you mean. It's very simply merging one branch into another. Features branch off of dev, merge back into dev. When it's time for a release, dev merges into master.

— Reply to this email directly or view it on GitHubhttps://github.com/FinalsClub/karmaworld/issues/279#issuecomment-32412629 .

[btbonval]

@sethwoodworth did you read the article? What I explained was what I had done at a previous workplace, which was an /extension/ of the article. We don't need to do what my previous workplace did, we don't have other clients or anything.

@charlesconnell Current example of where our current system is a headache: manage.py. I assure you I will commit my changes to manage.py into master no less than one time per week, with a followup "woops shit didn't mean to commit that AGAIN" commit

[charlesconnell]

There are some annoyances about our current setup that could be sidestepped with Bryan's suggested plan. I'm sure there will be new headaches introduced along the way, however. I'm okay with either one.

[btbonval]

@charlesconnell my hope is that those problems will come up far less often. I guarantee now that I have to have manage.py separate in my working space than the repo, I'm going to push my working space accidentally so much it's going to make me want to harm fish.

[sethwoodworth]

Yes, I've read the git-flow article. It's the canonical workflow for git IMO.

What do you mean by:

master branch can contain prod files.
dev branch can contain dev files.
vm branch can contain vm fies.

That is not what the article describes.

[sethwoodworth]

@charlesconnell Are you changing .dev to .prod? Why not add a production_manage.py file that has that change?

[btbonval]

You're right. The article does not describe that. @charlesconnell is running a different dev environment than the VM, but I'd like to standardize the dev env around the VM. There are some slight differences wherein the VM is more like production with a few bits of dev added and a few unique bits added.

@sethwoodworth we already have a crapton of needlessly repetitive files. You don't manage minor changes by copypasta, you do it with git branches.

[btbonval]

git branching is a fantastic way to adhere to DRY when it comes to settings and config.

[btbonval]

Quick research on settings and config. http://stackoverflow.com/questions/19771662/git-branching-model-workflow-develop-master-branches http://stackoverflow.com/questions/3214016/git-application-configuration-and-different-environments

Looks like people generally "compile" their settings and config external to git using templates stored inside git. Not a terrible approach. I'm willing to keep a single branch if we have config stubs which only store as much as is different for each environment. We need some way to compile our config prior to running Django.

[AndrewMagliozzi]

I am willing to give Bryan's approach a try. If it causes undo complexity, we can go back to simple feature branches. How about we give it two weeks then reassess. Everyone cool?

On Jan 15, 2014, at 4:04 PM, Bryan Bonvallet notifications@github.com wrote:

Quick research on settings and config. http://stackoverflow.com/questions/19771662/git-branching-model-workflow-develop-master-branches http://stackoverflow.com/questions/3214016/git-application-configuration-and-different-environments

Looks like people generally "compile" their settings and config external to git using templates stored inside git. Not at terrible approach. I'm willing to keep a single branch if we have config stubs which only store as much as is different for each environment. We need some way to compile our config prior to running Django.

— Reply to this email directly or view it on GitHub.

[btbonval]

The forked git branches never happened. It seemed too difficult to put a freeze on coding to rejigger everything, and potentially the end result would have been too difficult for folks newer to git.

The real problem here is maintaining configuration in source code.

If we move to Heroku as desired, then dev, staging, and prod will all be run on Heroku (consistent environment). It'd be great if we could make use of Heroku config vars. https://devcenter.heroku.com/articles/config-vars "A given codebase may have numerous deployments: a production site, a staging site, and any number of local environments maintained by each developer. An open source app may have hundreds or thousands of deployments.

Although all running the same code, each of these deploys have environment-specific configurations. One example would be credentials for an external service, such as Amazon S3. Developers may share one S3 account, while the staging site and production sites each have their own keys.

The traditional approach for handling such config vars is to put them under source - in a properties file of some sort. This is an error-prone process, and is especially complicated for open source apps which often have to maintain separate (and private) branches with app-specific configurations.

A better solution is to use environment variables, and keep the keys out of the code. On a traditional host or working locally you can set environment vars in your bashrc. On Heroku, you use config vars."

[AndrewMagliozzi]

I'm a fan of ENV variables or config vars

On Wed, May 14, 2014 at 3:51 PM, Bryan Bonvallet notifications@github.comwrote:

The forked git branches never happened. It seemed too difficult to put a freeze on coding to rejigger everything, and potentially the end result would have been too difficult.

The real problem here is maintaining configuration in source code.

If we move to Heroku as desired, dev, staging, and prod will all be run on Heroku (consistent environment). It'd be great if we could make use of Heroku config vars. https://devcenter.heroku.com/articles/config-vars "A given codebase may have numerous deployments: a production site, a staging site, and any number of local environments maintained by each developer. An open source app may have hundreds or thousands of deployments.

Although all running the same code, each of these deploys have environment-specific configurations. One example would be credentials for an external service, such as Amazon S3. Developers may share one S3 account, while the staging site and production sites each have their own keys.

The traditional approach for handling such config vars is to put them under source - in a properties file of some sort. This is an error-prone process, and is especially complicated for open source apps which often have to maintain separate (and private) branches with app-specific configurations.

A better solution is to use environment variables, and keep the keys out of the code. On a traditional host or working locally you can set environment vars in your bashrc. On Heroku, you use config vars."

— Reply to this email directly or view it on GitHubhttps://github.com/FinalsClub/karmaworld/issues/279#issuecomment-43128854 .

[charlesconnell]

Since we're doing config management with environment variables now, I'm closing this ticket.

[btbonval]

Yay! We should have been doing this all along :(