fabric8's own release pipeline

[rawlingsj]

New release process

I've been working this week on changing the fabric8 release process to fully embrace the Jenkins 2.0 approach towards CD

What does this mean..

• adopt a better Continuous Deployment / Continuous Delivery approach
• automated releases rather than manually kicking off releases
• each fabric8 project or microservice is responsible for it's own release pipeline so will have its own Jenkinsfile
• all be aware of the process and make suggestions + automated tests where we can

I'm also proposing we move to a Continuous Deployment (automatic release on every commit to master) for..

• kubernetes-model
• kubernetes-client
• fabric8
• ipaas-quickstarts
• fabric8-devops
• fabric8-ipaas
• kansible
• kubeflix
• portswizzler

This may sound drastic but it will help us over time improve our CI process to ensure changes that get committed to master are ready for release. As it stands at the moment there's no extra testing we carry out after a change is merged to master for the projects above so we might as well release quickly to get quicker feedback if there's a problem.

Continuous Delivery with staging to a test environment + human approval for

• fabric8-console
• fabric8-forge <- this includes staging the CD-PIPELINE template in a test env

@rhuss take a look at these projects when they're running and have a think about how you might like the d-m-p pipeline to be.

Additional changes..

Maven and CD is a PITA. We typically use the pom SNAPSHOT version to get the next release version, once the project has been released we would normally increment the next snapshot version and push that change back to Github. This causes problems during continuous deployment because the pom update commit would trigger another release creating an unwanted loop.

Going forward we're going to try using the Git tag during the release to figure out the next release version, once we have the new version push that new tag back to GitHub. Subsequent changes will get a new release version and run in parallel even if the previous release is still in its pipeline (i.e. being tested / waiting for approval etc). @chirino @jstrachan this will stop the merge issues you've been seeing to the product branches.

Other points..

We now have a CD test project that performs a real release of artifacts to maven central, build and pushing docker image to dockerhub in order to test changes to our Jenkins Global Library (the reusable groovy functions). Eventually it would be nice to automate testing using this project when we push changes to https://github.com/fabric8io/jenkins-workflow-library

Next steps..

• roll this out to the projects above, I'd like to start this straight away
• Chaining pipelines need to figure out a way to chain multiple Pipelines together i.e. at the moment we have a single Jenkinsfile that orchestrates the release of each project then updates the version number of a dependency and makes a pull request. For example when release=ing kubernetes-model, when its artifacts are available in maven central the pipeline automatically updates the <kubernetes-model.version> in kubernetes-client, submits a PR and waits until merged before releasing kubernetes-client. This wont happen now as we move away from this global Jenkinsfile and each project has its own Jenkinsfile, so maybe once kubernetes-model is available in maven central the kubernetes-model/Jenkinsfile should make pull requests on any project that use it which would then result in a release for that project. For now I'll hardcode in the Jenkinsfile which projects need to be updated but going forward we should maybe scan the fabric8 repos looking for any poms that needs updating?
• End to end Pipeline visualisation as each project will now have its own pipeline there wont be the overall pipeline visualisation that covers all the releases from kubernetes-model right through to fabric8-forge

[jstrachan]

so how about we do, say, every day to start with a 'canary' release where we iterate through all the projects and

• create a new version number
• create a canary branch for this new version
• check if any dependencies have changed and update the poms etc
• create a release branch, update the project's actual version
• do a mvn deploy / docker build to local nexus and docker registry

when everything is built we run integration tests (e.g. starting cd-pipeline up in a new namespace, does it all start - then we try use Selenium to create a few projects and check they build and provision)

Then we wait on human approval, then promote.

To start with we don't need the promote - just do it as a nightly test that everything's good & send an email with a detailed test report of who changed what, what the tests results were etc.

Ideally eventually we'd be able to promote though; by moving the local nexus build to sonatype; pushing to docker hub and merging the canary dependency changes up to master branch

[rawlingsj]

Ok so what I've got at the moment is two test projects, both in separate git repos, the second has a maven dependency on the first.

Each project can be released independently from each but ee also want to perform a multi project release that is built, staged, tested and promoted together. In more detail..

• check for dependency updates
• make github pull request (but dont merge)
• perform a canary release
• stage them to sonartype (OSSRH)
• stage docker images
• deploy to a test environment
• approve
• release both projects in OSSRH staging repo together
• promote docker images to dockerhub
• merge pull requests that update dependency versions.

This I believe is now done, I've refactored the global workflow library release scripts, as well as created a separate git repo https://github.com/fabric8io/fabric8-release-pipelines which contains a Jenkinsfile in different branches when we want to release multiple projects together (i.e. not a single release). Both types of Jenkinsfile describe the pipeline but also reference a release.groovy that sits in the source code with each project. The release.groovy is then reused by the single Jenkinsfile when we want a single release, as well as any multi project Jenkinsfiles when they are part of a bigger release.

This is the Jenkins stage view of the above pipeline on the test projects..

There's still a little bit of work to get a staged fabric8 release running in a test environment, and also the release.groovy files aren't very DRY as there's property repetition but I think I'm ready to apply this to a couple of the fabric8 projects. I'll probably start with fabric8-devops and fabric8-forge so I can get the test environment and approval tied in.

What's nice is we should be able to soon import fabric8 projects into fabric8, so we can take advantage of the pipeline + environments dashboards, see the git commit changes, links to test environments, kibana dashboards etc.

Test project 1 - https://github.com/fabric8io/pipeline-test-project-dependency Test project 2 - https://github.com/fabric8io/pipeline-test-project