Update Process_guide.md

[kkellerlbl]

This mostly dummy edit is a placeholder to try to discuss the workflow we want to use for branches and deploying those branches to our various environments.

[kkellerlbl]

There was a fairly long email discussion about which workflows we might want to use. Here are some of the links which came up.

https://www.atlassian.com/git/tutorials/comparing-workflows http://nvie.com/posts/a-successful-git-branching-model/ (the "gitflow workflow" proposed by Atlassian) http://drewfradette.ca/a-simpler-successful-git-branching-model/ (a simplification of the above workflow) https://github.com/kbase/narrative/blob/master/docs/git-workflow.md (an adaptation of the Fradette model)

[samseaver]

Hello, so reading some of this made me remember one problem I do have with the fork-and-pull process, particularly if you desire each PR to be "atomic" in its nature so to document separate changes etc.

There has been several times in the past when I made several different changes in the code, and create several different commits, all relatively close together in time. Each time, I made a pull-request, I then found that I couldn't separate out the commits into different pull-requests, and if my pull-request was left un-merged for some amount of time, even weeks, then each subsequent commit would have to go into the same pull-request.

I found that, since a pull-request on the whole compares your entire branch with that in the parent repository, you can't actually therefore separate out individual pull-requests, so, it appears to me that the only logical approach would then be to create a separate branch for each change you want to make that you believe should have its own pull-request, is that what we're advocating?

[samseaver]

As a follow-up, I read some of the other pages linked on Git branching models, and clearly, as well as discussing the proper use of pull-requests we should discuss the proper use of branching. It seems appropriate to me that individual branches should be created by a developer in order to address individual features or hotfixes etc. and that none of these branches would be stored on origin.

It then follows that a developer is responsible for the maintenance of their own branches, including their deletion, as they proliferate, but it would help to come up with a process guide for branch maintenance.

[scanon]

I think having a separate private branch for each change is the way to go. A little tedious at times, but the most bullet proof.

[mlhenderson]

The 3 branch model of stable, develop, release seems reasonable to me, but if there are other view points it would be useful to discuss the trade offs.

If the master branch is matched to what is in production, the develop branch can be continuously used to develop against, with a release branch used to capture a snapshot of development poised for testing and deployment that can be patched. In this model, we would have develop running in CI, release running in nextgen, and then master running in production. @kkellerlbl Is that right?

[kkellerlbl]

Mostly. For narrative and some other repos, develop is currently deployed on -ci, the staging branch (we renamed it because I thought calling the branch that wasn't released to production "release" was strange) is on -next (not really nextgen), and master on production.

I've seen some workflows where master is used in a CI-like environment instead of a separate develop branch, and then there's a separate named branch that's deployed to production (I've seen it called "release" or "deploy"). I wasn't a big fan of this because I feel like master is the branch people are likely to go to first, and the first thing we should present to the public is what we have in production.

[samseaver]

So, based on what Keith says, we're not using narrative-dev?

On Thu, Apr 30, 2015 at 6:00 PM, kkellerlbl notifications@github.com wrote:

Mostly. For narrative and some other repos, develop is currently deployed on -ci, the staging branch (we renamed it because I thought calling the branch that wasn't released to production "release" was strange) is on -next (not really nextgen), and master on production.

I've seen some workflows where master is used in a CI-like environment instead of a separate develop branch, and then there's a separate named branch that's deployed to production (I've seen it called "release" or "deploy"). I wasn't a big fan of this because I feel like master is the branch people are likely to go to first, and the first thing we should present to the public is what we have in production.

— Reply to this email directly or view it on GitHub https://github.com/kbase/project_guides/pull/43#issuecomment-97994732.

Postdoctoral Fellow Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Avenue Argonne, IL 60439

http://www.linkedin.com/pub/sam-seaver/0/412/168 samseaver@gmail.com (773) 796-7144

"We shall not cease from exploration And the end of all our exploring Will be to arrive where we started And know the place for the first time." --T. S. Eliot

[kkellerlbl]

So...narrative-dev. It's a bit of a special case, and I don't think we've totally settled on how to use it (if people have concrete suggestions let's make a pull request for it). Here's what I imagine we can do; this is not meant to be binding, it's just one suggestion.

I think narrative-dev can be used for development of the backend elements of the narrative, like the lua code (and/or alternative ways of facilitating communication between nginx and Docker), the spinning up and shutting down of containers, logging, fiddling with different instances of other backend services, probably other things as well. Tools like Vagrant can help with this but sometimes it's hard to exactly replicate the KBase environment in a Vagrant instance. IOW it could be a special sort of development sandbox, sort of like a Magellan instance.

In contrast, I can see -ci being used more for testing out code that a module author is hoping to get deployed to production (including the backend items I talked about above). So -ci could be used for validation (both manual, and when we have them automated tests) of code. Under this proposal, I would suggest that narrative-dev not be used as the basis for a decision on whether anything should be deployed to production-testing could still take place there, but modules would still have to be tested in -ci before being approved for production (or -next) deployment.

I hope that makes sense; I feel like I'm stumbling over what I have in my head. Feel free to follow up if I'm still not explaining well.

I want to reiterate that this is just one idea that I have, and that we're not bound by it. I'm open to other ideas for using narrative-dev.

[samseaver]

Right, so if one were to develop a new feature, they'd fork the repo, create a new branch from 'develop', commit, PR, then at various stages the merged PRs in 'develop' would be merged to 'staging', and then to 'master'.

Do we have a schedule in mind for how regularly staging would be updated from develop, and would one be allowed to branch directly from staging in order to apply a fix, or would all fixes have to go through 'develop'?

On Thu, Apr 30, 2015 at 11:38 PM, kkellerlbl notifications@github.com wrote:

So...narrative-dev. It's a bit of a special case, and I don't think we've totally settled on how to use it (if people have concrete suggestions let's make a pull request for it). Here's what I imagine we can do; this is not meant to be binding, it's just one suggestion.

I think narrative-dev can be used for development of the backend elements of the narrative, like the lua code (and/or alternative ways of facilitating communication between nginx and Docker), the spinning up and shutting down of containers, logging, fiddling with different instances of other backend services, probably other things as well. Tools like Vagrant can help with this but sometimes it's hard to exactly replicate the KBase environment in a Vagrant instance. IOW it could be a special sort of development sandbox, sort of like a Magellan instance.

In contrast, I can see -ci being used more for testing out code that a module author is hoping to get deployed to production (including the backend items I talked about above). So -ci could be used for validation (both manual, and when we have them automated tests) of code. Under this proposal, I would suggest that narrative-dev not be used as the basis for a decision on whether anything should be deployed to production-testing could still take place there, but modules would still have to be tested in -ci before being approved for production (or -next) deployment.

I hope that makes sense; I feel like I'm stumbling over what I have in my head. Feel free to follow up if I'm still not explaining well.

I want to reiterate that this is just one idea that I have, and that we're not bound by it. I'm open to other ideas for using narrative-dev.

— Reply to this email directly or view it on GitHub https://github.com/kbase/project_guides/pull/43#issuecomment-98042054.

Postdoctoral Fellow Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Avenue Argonne, IL 60439

http://www.linkedin.com/pub/sam-seaver/0/412/168 samseaver@gmail.com (773) 796-7144

"We shall not cease from exploration And the end of all our exploring Will be to arrive where we started And know the place for the first time." --T. S. Eliot

[kkellerlbl]

I'll talk about schedule first: one thing Fernando and I were tossing around (again, I want to stress, not as a policy statement, just as an idea, and this one not really fleshed out) was that a repo would be eligible to push its code to staging, and thus be deployed in -next, once it had passed rigorous tests in -ci, and the authors requested it. I don't even know if that's a good idea, or whether we should instead have a set schedule. But one way or another, if we were to adopt this model, then changes could not be merged from develop to staging unless they'd passed a lot of testing in -ci, and could not be merged from staging to master unless they'd passed testing in -next.

I think that your suggested workflow is basically reasonable within this workflow model. If the developer has write access to the repo he can also create a feature branch directly on the repo, and do PRs from there instead. But I believe the two processes are basically morally equivalent.

[samseaver]

Maybe the idea of testing in narrative-ci and merging/deploying to narrative-next will work on an individual basis, if there's several features being developed and tested, I imagine some temporal co-ordination would be needed to avoid headaches, but it also seems to me that approaching a major production release, a fixed schedule, as activity increases, would be beneficial for all parties.

On your last note, I don't think you can do a PR for commits made directly to a repo, you'd have to fork it first, unless I mis-understand you.

On Fri, May 1, 2015 at 10:31 AM, kkellerlbl notifications@github.com wrote:

I'll talk about schedule first: one thing Fernando and I were tossing around (again, I want to stress, not as a policy statement, just as an idea, and this one not really fleshed out) was that a repo would be eligible to push its code to staging, and thus be deployed in -next, once it had passed rigorous tests in -ci, and the authors requested it. I don't even know if that's a good idea, or whether we should instead have a set schedule. But one way or another, if we were to adopt this model, then changes could not be merged from develop to staging unless they'd passed a lot of testing in -ci, and could not be merged from staging to master unless they'd passed testing in -next.

I think that your suggested workflow is basically reasonable within this workflow model. If the developer has write access to the repo he can also create a feature branch directly on the repo, and do PRs from there instead. But I believe the two processes are basically morally equivalent.

— Reply to this email directly or view it on GitHub https://github.com/kbase/project_guides/pull/43#issuecomment-98157449.

Postdoctoral Fellow Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Avenue Argonne, IL 60439

http://www.linkedin.com/pub/sam-seaver/0/412/168 samseaver@gmail.com (773) 796-7144

"We shall not cease from exploration And the end of all our exploring Will be to arrive where we started And know the place for the first time." --T. S. Eliot

[scanon]

I agree. My thought is we would some regular schedule when we would start to gear up for a release. Ideally this would mean we merge some version of devel to staging, then we would do incremental changes to that branch to get it solid for release. Under the master model, we would branch for the release and the small changes for the official release would occur along that branch. Under either model, any bug fixes done to prepare the release would likely get applied to both the staging branch and the develop branch.

—Shane

On May 1, 2015, at 8:42 AM, samseaver notifications@github.com wrote:

Maybe the idea of testing in narrative-ci and merging/deploying to narrative-next will work on an individual basis, if there's several features being developed and tested, I imagine some temporal co-ordination would be needed to avoid headaches, but it also seems to me that approaching a major production release, a fixed schedule, as activity increases, would be beneficial for all parties.

On your last note, I don't think you can do a PR for commits made directly to a repo, you'd have to fork it first, unless I mis-understand you.

On Fri, May 1, 2015 at 10:31 AM, kkellerlbl notifications@github.com wrote:

I'll talk about schedule first: one thing Fernando and I were tossing around (again, I want to stress, not as a policy statement, just as an idea, and this one not really fleshed out) was that a repo would be eligible to push its code to staging, and thus be deployed in -next, once it had passed rigorous tests in -ci, and the authors requested it. I don't even know if that's a good idea, or whether we should instead have a set schedule. But one way or another, if we were to adopt this model, then changes could not be merged from develop to staging unless they'd passed a lot of testing in -ci, and could not be merged from staging to master unless they'd passed testing in -next.

I think that your suggested workflow is basically reasonable within this workflow model. If the developer has write access to the repo he can also create a feature branch directly on the repo, and do PRs from there instead. But I believe the two processes are basically morally equivalent.

— Reply to this email directly or view it on GitHub https://github.com/kbase/project_guides/pull/43#issuecomment-98157449.

Postdoctoral Fellow Mathematics and Computer Science Division Argonne National Laboratory 9700 S. Cass Avenue Argonne, IL 60439

http://www.linkedin.com/pub/sam-seaver/0/412/168 samseaver@gmail.com (773) 796-7144

"We shall not cease from exploration And the end of all our exploring Will be to arrive where we started And know the place for the first time." --T. S. Eliot — Reply to this email directly or view it on GitHub https://github.com/kbase/project_guides/pull/43#issuecomment-98161170.

[kkellerlbl]

Sam, yes, there might need to be some coordination if there are many moving components that we want promoted from -next to -ci at the same time. I've brought up this scenario in some conversations here: say repo X wants to move to -ci, and has passed all of its automated and manual testing there. Then X/develop gets merged into X/staging and deployed to -next, but it turns out it needs a feature from Y/develop, which isn't on staging yet.

I think that there are workflows which have methods for managing this situation, but I haven't had time to look for them yet.

There's no a priori requirement to commit to a "major release" cycle: in a continuous delivery model, for example, a new release is pushed out when it's ready. With really good testing this is feasible, but I doubt we're ready for that model. (And we might not want to adopt CD anyway; I don't have enough information to decide whether I like CD or not.)

You can do a PR from one branch in a repo to another branch in the same repo: that's basically how all our PRs for the project_guides have been working. For example this PR is the workflow-discussion branch of kbase/project_guides. For edits I make directly on Github (fine for markdown), Github actually automatically determines whether your edit can be done on a branch or whether it needs to be forked, which is pretty cool. For real code you wouldn't do this obviously. :) But you could attempt to create the branch directly from Github, and then fork if you didn't have write permission.

[fperez]

OK, there was good discussion here, but the actual content of the PR seems uncontroversial, so I'm going to merge it to clear the backlog. Thanks!