search

opendevstack / ods-jenkins-shared-library

Shared Jenkins library which all ODS projects & components use - provisioning, SonarQube code scanning, Nexus publishing, OpenShift template based deployments and repository orchestration
Apache License 2.0
73 stars 57 forks source link

Decide branching and deployment workflow #21

Closed michaelsauter closed 5 years ago

michaelsauter commented 6 years ago

Currently, the branching and deployment workflow is a mess and difficult to understand.

After discussing this with @clemensutschig, here is the proposed solution.

The goal is to allow each team using the shared library to pick a standardized, easy to understand workflow that fits to the teams needs.

There are two main ways to work:

  1. Focus on continuous integration

This is backed by gitflow. develop is the integration branch, and feature/bugfix branches target this branch. master is the production branch, typically updated at the end of every sprint. Additionally, release-x.x.x branches can be forked from develop to ensure quality before the release is promoted to production.

In regards to OpenShift, master is mapped to the test project, and develop is mapped to the dev project. Projects for PR branches and release branches can be created as required by the team (e.g. to support UAT).

  1. Focus on continuous deployment

This is backed by GitHub flow. master is the default branch and feature/bugfix branches target this branch. Each pull request is promoted to production immediately, with no integration branch in between. This style is useful for fast development, and/or when a good review system is in place.

In regards to OpenShift, master is mapped to the test project, and PR branches are either deployed onto the dev project or a specific project created just for the PR.

How does that sound? Please - everyone - provide feedback and ask questions if something is not clear.

@stitakis @gerardcl @lukas-kloeppel @renedupont @m-apsolon @hugowschneider @adyachok

hugowschneider commented 6 years ago

In my option, focus on continuous deployment is what we are trying to achieve and should be the focus.

In regards to OpenShift, master is mapped to the test project, and PR branches are either deployed onto the dev project or a specific project created just for the PR.

I would make the environment per branch mandatory. It would allow a good review process to take place within a running environment and remove the race condition in dev project.

michaelsauter commented 6 years ago

@hugowschneider Thanks for feedback.

I think there are reasons why (esp. during MVP) one would not want the overhead of deploying per-branch OCP projects (save resources, shorten build time, simpler data management, ...).

But if I read your message correctly, then how you would like to work would fit into the second process (GitHub flow), right?

Regarding OCP project creation, I think one could have two settings to control:

  1. autoCreateReleaseEnvironments: Only relevant for the gitflow, it would create an OCP environment for every branch starting with release-, so e.g. branch release-1.1.0 would be deployed to foo-rel-1.1.0 automatically.

  2. autoCreateStoryEnvironments: Relevent for both gitflow and GitHub flow, it would create an OCP environment for branches that contain a JIRA user story, e.g. feature/foo-375-do-this would be deployed to foo-dev-365.

Or ... manual creation of environments as teams see fit.

m-apsolon commented 6 years ago

[edit] this message was meant to be a response to @hugowschneider

Please refrain from making these overhead decisions for every team @ oc env for every story. Our team for one would like to skip this in our current phase.

On Tue, Aug 28, 2018, 13:44 Michael Sauter notifications@github.com wrote:

@hugowschneider https://github.com/hugowschneider Thanks for feedback.

I think there are reasons why (esp. during MVP) one would not want the overhead of deploying per-branch OCP projects (save resources, shorten build time, simpler data management, ...).

But if I read your message correctly, then how you would like to work would fit into the second process (GitHub flow), right?

Regarding OCP project creation, I think one could have two settings to control:

1.

autoCreateReleaseEnvironments: Only relevant for the gitflow, it would create an OCP environment for every branch starting with release-, so e.g. branch release-1.1.0 would be deployed to foo-rel-1.1.0 automatically. 2.

autoCreateStoryEnvironments: Relevent for both gitflow and GitHub flow, it would create an OCP environment for branches that contain a JIRA user story, e.g. feature/foo-375-do-this would be deployed to foo-dev-365.

Or ... manual creation of environments as teams see fit.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opendevstack/ods-jenkins-shared-library/issues/21#issuecomment-416553946, or mute the thread https://github.com/notifications/unsubscribe-auth/AFfyzML9JwD7xkbJi0yEZjzjsk6r2VTNks5uVS0FgaJpZM4WPFZ1 .

renedupont commented 6 years ago

I guess the idea is rather to set these settings optionally. So if you don't set them, no oc envs will be created for any branch besides master and develop.

michaelsauter commented 6 years ago

@m-apsolon Sure, I agree that forcing a separate OC env for every story wouldn't be good. That should be an opt-in feature. I think for teams using gitflow, it is less likely to be used - and for teams using GitHub flow more likely.

michaelsauter commented 6 years ago

Adding to my initial proposal, here's a more detailed explanation of the semantics:

Teams can set either config.workflow: "gitflow" or config.workflow: "githubflow". My suggestion would be to make githubflow the default as it is simpler and works out-of-the-box (for gitflow one needs to create a develop branch and change settings in BitBucket).

For config.workflow: "githubflow", the following applies:

For configworkflow: "gitflow", the following applies:

In addition, one can opt-in to automatic creation of environments:

Regarding defaults, one could also argue that it would be good to have config.autoCreateReleaseEnvironment: true for config.workflow: "gitflow" and config.autoCreateReviewEnvironment: true for config.workflow: "githubflow". Opinions?

michaelsauter commented 6 years ago

FYI @ferhatozkan @ramashah @sino92 @rianet

m-apsolon commented 6 years ago

@michaelsauter your proposal sounds good to me :)

lukas-kloeppel commented 6 years ago

@michaelsauter The proposal sounds good. I think these both flows cover most of the use cases users will have.

Also the defaults of config.autoCreateReleaseEnvironment: true for config.workflow: "gitflow" and config.autoCreateReviewEnvironment: true for config.workflow: "githubflow" make totally sense for me, especially when starting with business testing before the release.

We could also add the settings config.autoDeleteReleaseEnvironmentOnMerge and config.autoDeleteReviewEnvironmentOnMerge which will delete the full environment when it is merged into master (Do we have the required data available in Jenkins to identify the merge?)

stitakis commented 6 years ago

Hi all, although some details still need to be explored and well documented, for me, latest Michael proposal describes very well key concepts of 2 options. To get to these details I think it would help to go thru the different use case scenarios (like how to create a dev branch, how to create a release, how to stage a release to prod branch, etc...) and write short how to like a recipe. Doing so, we could figure out important implementation details.

michaelsauter commented 6 years ago

@stitakis Thanks for feedback. I am not entirely sure what "recipes" you are after, since both gitflow and Github flow describe the workflow well on their websites and we just follow the instructions there? To me, implementation sounds (relatively) straightforward as actually only very few things need to change compared to what we have now. Or did you mean sth. else?

The one area where it would be good to have documentation for is how to setup BitBucket to work well with gitflow (as in, instructions how to configure the develop branch correctly).

Another area that might cause issues is how to ensure things don't break between OpenShift environments. For example, imagine you are working with gitflow and you have a release-1.0.0 branch that is currently deployed into release-1.0.0 on OCP. Now you want to promote this branch, so you merge release-1.0.0 into master, which consequently gets deployed to OCP env test. At this point, there is no guarantee that the deployment works correctly because you might have made modifications to your infrastructure in the release-1.0.0 environment, which are not automatically transferred to the test environment. That said, I think those kinds of issues (you will have similar ones in the Github flow) are a little bit out of scope here, and solving them is non-trivial.

@lukas-kloeppel I think this was discussed earlier and it is not easy to know when it is ok to remove an environment, as for that you'd need to know if the JIRA user story was accepted.

stitakis commented 6 years ago

@michaelsauter: yes the websites provides good instruction. No question on that... ....I was thinking more about how to deal with Openshift, about things like how to do hotfix in production once development moved forward or similar, or what you mentioned about how to don´t break between Openshift environments. This is not a trivial sometimes.

For me, the root of complexity for this changes is how to map git "events" to openshift. In my project we reduced the number of openshift enviroments to 4. Changes are promoted with PRs across this branches dev, master, uat and prod (which are mapped to OCP environments: (dev, test, uat and prod). It is kind of incremental deployment strategy.

So far, this strategy is working good. We added in Jenkinsfile some logic that setup the config map and secret automatically. This reduced a lot the effort of setting config maps and secret manually.

michaelsauter commented 6 years ago

@stitakis Actually, how does your current setup correspond to the suggestion here? You have dev, test, uat, prod. But in my suggestion, you would not have all of those. In my suggestion, test is based on master, so it is effectively your production environment. Further, there is no uat, but I guess that is equivalent to the release- environments?

(This point also leads to the question whether we should rename the test environment to prod, which is the more appropriate name really ...)

I know you are using Jenkins to setup config maps / secrets automatically, and I think this is one good way to solve some of the problems when switching between environments, but I don't think that it is part of the discussion here - we should discuss separately if/how the shared library should support the user in those questions.

The other point you mention is hotfixes for production. That is a good point for gitflow, I need to add that to the mapping. I'm going to create a table that shows the mapping ...

rianet commented 6 years ago

@michaelsauter master branches should always be our prod environments, makes sense to rename.

michaelsauter commented 6 years ago

Here's a branch-to-environment mapping. The decision is made top-to-bottom: first match wins.

Branch git-flow GitHub Flow
master test test
develop dev (no special handling, continue with next row)
hotfix/*-123-*, where 123 is a JIRA user story ID review-123 if it exists or is auto-created (cloned from test) (no special handling, continue with next row)
release/x.y.z release-x.y.z if it exists or is auto-created (cloned from dev) (no special handling, continue with next row)
*-123-*, where 123 is a JIRA user story ID review-123 if it exists or is auto-created (cloned from dev) review-123 if it exists or is auto-created (cloned from test), otherwise dev
* - -

Whether or not an environment is auto-created is controlled by two configurations:

If there are no objections to these semantics, I'll take a stab at implementing them next week.

stitakis commented 6 years ago

@michaelsauter it looks very well for me. I think we could adopt the git-flow branch-to-environment in my project.

As brainstormed today... to minimize environment creation and setup effort it would be good to implement the option to deploy to a default target environment. Let say for git-flow for release/x.y.z the target environment would be release (or some other name). For hotfix/*-123* would be uat (or pre). It would be really helpful too in case auto-created=false to create the target environment in case release and uat envs do not exits. What do you think?

lukas-kloeppel commented 6 years ago

@michaelsauter I would also vote to change the name from test environment to prod, this would be a better naming, as you already suggested

m-apsolon commented 6 years ago

Lukas, if we did that, we would save so many meaningless arguments and brain power! My Santa's wishlist no 1

On Tue, Sep 4, 2018, 10:22 Lukas Kloeppel notifications@github.com wrote:

@michaelsauter https://github.com/michaelsauter I would also vote to change the name from test environment to prod, this would be a better naming, as you already suggested

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/opendevstack/ods-jenkins-shared-library/issues/21#issuecomment-418282876, or mute the thread https://github.com/notifications/unsubscribe-auth/AFfyzJbYMd9hkCzb5hTf52IAquV2iNS6ks5uXjgtgaJpZM4WPFZ1 .

michaelsauter commented 5 years ago

Finally, what has been discussed here has now been merged into master: #25.

However, the semantics have evolved quite a bit - we now expose the mapping between branches and OCP environments directly, instead of controlling it via a workflow setting. This is both easier to understand and more flexible. See the readme for details.

Unfortunately, the test env will continue to be named test for reasons :) However, you can create a new prod environment and map one of your branches to that ...