search

griddynamics / mpl

[IT-36925] Jenkins Shared Modular Pipeline Library
https://blog.griddynamics.com/developing-a-modular-pipeline-library-to-improve-devops-collaboration/
Apache License 2.0
156 stars 97 forks source link

readme

Modular Pipeline Library

CircleCI Gitter

CircleCI nightly LTS - testing MPL pipeline with the current LTS Jenkins every night

CircleCI nightly Latest - testing MPL pipeline with the current Latest Jenkins every night

Shared jenkins library with modular structure allow to write a simple pipeline modules, test it properly and use in any kind of pipelines.

Goals

Documentation

This readme contains mostly technical information, if you need some overview - please check the next resources:

You also can check MPL Wiki to find additional info.

Dependencies

Setup

Go to: Manage Jenkins --> Configure System --> Global Pipeline Libraries:

Usage

You can use MPL in 3 different ways - it's 3 layers of the library:

Jenkinsfile / Pipeline script

Just two lines to use default Master pipeline in your project Jenkinsfile or in the Jenkins Pipeline Script:

@Library('mpl') _
MPLPipeline {}

Pipelines

You can find pipelines in the MPL library interfaces: {MPL}/vars/MPL{Name}Pipeline.groovy files.

There is 2 ways to slightly reconfigure the provided standard pipeline:

If both ways are not helping - just create your own pipeline definition in the Jenkinsfile and use MPL modules or create your own nested library with your standard pipelines.

Configuration

Usually configuration is initialized in the pipeline - it's calling MPLPipelineConfig interface with arguments:

After that pipeline defining MPL object and use it's common functions to define the pipeline itself. Pipeline is calling MPLModule that calling the required module logic.

In the module we have a common predefined variables (like default steps, env, params, currentBuild...) and a new variable contains the pipeline/module configs: CFG. It's a special MPLConfig object that defines interface to get and set the properties. It's promising a number of things:

MPLConfig object is not the usual Map - you have to use it with known key you defined before. That was introduced to make sure you will see what kind of configuration is used or required on each stage. So you will need to use it as def a = CFG.really_needed_data instead of just def a = CFG or for( a in CFG ) println(a) which will tell you much more useful information and simplify debug.

Use of the CFG object is quite simple. Imagine we have the next pipeline configuration:

[
  agent_label: '',
  val1: 4,
  val2: [
    val_nested: 'value',
    val_list: [1,2,3,4],
  ],
]

So you got the point - hopefully this will be helpful and will allow you to create the great interfaces for your modules.

MPLModule return

MPLModule step running the specified module with CFG configuration and returns OUT configuration. OUT is always empty when a module just started and could be modified inside the module. So:

To modify the pipeline config with the module output - just use MPLPipelineConfigMerge step - we recommend to use it only in the pipeline step specification to concentrate any pipeline-related changes in the pipeline definition itself.

You can use MPLPipelineConfigMerge in the pipeline like this - the logic will put artifact key with value [version: 1] in the global configuration and you will be able to use `CFG.'artifact.version' in the following modules:

pipeline {
    ...
    steps {
        MPLPipelineConfigMerge(MPLModule('Some Module').artifact_info)
    }
    ...

Modules

MPL is mostly modules with logic. Basic features:

In fact modules could be loaded from a number of places:

If you will override module Build in your project repo, it will be used first. If you will try to require Build module from this overridden Build module - original library module will be used.

Check the usage examples & library modules to get some clue about the nesting system.

Creating / Overriding steps modules

If your project is special - you can override or provide aditional modules just for the project.

What do you need to do:

  1. Create a step file: {ProjectRepo}/.jenkins/modules/{Stage}/{Name}{Stage}.groovy (name could be empty)
  2. Fill the step with your required logic: you can use CFG config object & MPL functions inside the steps definition
  3. Use this step in your custom pipeline (or, if it's override, in standard pipeline) via MPLModule method.
  4. Provided custom modules will be available to use after the checkout of your project source only

For example: "Maven Build" steps have path modules/Build/MavenBuild.groovy and placed in the library - feel free to check it out.

Post Steps

MPL supports 2 useful poststep interfaces which allow you to store all the logic of module in the same file.

All the poststeps will be executed in LIFO order and all the exceptions will be collected & displayed in the logs.

MPLModulePostStep

Allow to set some actions that need to be executed right after the current module (doesn't matter it fails or not).

Could be useful when you need to collect reports or clean stage agent before it will be killed.

If module post step fails - it's fatal for the module, so the pipeline will fail (unlike general poststeps). All the poststeps for the module will be executed and errors will be printed, but module will fail.

MPLPostStep

General poststep interface usually used in the pipelines. Requires 2 calls - first one to define poststep in a module and second one to execute it and usually placed in the pipeline post actions.

When error occurs during poststeps execution - it will be printed in the log, but status of pipeline will not be affected.

  1. {NestedLibModules}/Deploy/OpenshiftDeploy.groovy:

    MPLPostStep('always') {
echo "OpenShift Deploy Decomission poststep"
}

echo 'Executing Openshift Deploy process'
  2. {NestedLib}/var/CustomPipeline.groovy: 
    def call(body) {
// ...
pipeline {
  // ...
  stages {
    // ...
    stage( 'Openshift Deploy' ) {
      steps {
        MPLModule()
      }
    }
    // ...
  }
  post {
    always {
      MPLPostStepsRun('always')
    }
    success {
      MPLPostStepsRun('success')
    }
    failure {
      MPLPostStepsRun('failure')
    }
  }
}
}

Enforcing modules

To make sure that some of your stages will be executed for sure - you can add a list of modules that could be overrided on the project side. Just make sure, that you executing function MPLEnforce and provide list of modules that could be overriden in your pipeline script:

Notices:

Nested libraries

MPL supporting the nested libraries to simplify work for a big teams which would like to use MPL but with some modifications.

Basically you just need to provide your vars interfaces and specify the mpl library to use it:

And after that and configuring the library for your jenkins (just put it after the mpl config) - you can use it in the project's Jenkinsfile (see examples).

Also you can override resources of the MPL library - but it's forever. You can't use MPL resource anymore if you will override it in the nested library.

You can cover the nested library with tests as well as MPL library - please check the nested library example on wiki page.

Release process

Jenkins shared libraries is just a repositories connected to the Jenkins Master instance. So you can use any branch/tag from the MPL or nested lib repo.

Examples

Wiki page examples

Please check the wiki page to see some MPL examples: MPL Wiki

Standard Pipeline usage

If we fine with standard pipeline, but need to slightly modify options.

Use Standard Pipeline but with custom module

We fine with standard pipeline, but would like to use different deploy stage.

Custom Declarative Pipeline with mixed steps

Using nested library (based on MPL)

Contribution

Tests

We should be ensure that modules will not be broken accidentally, so tests is mandatory.

MPL supports MPLModule testing via slightly modified JenkinsPipelineUnit. You can find module tests (as well as modified base test classes & overridden requirements classes) in the test directory.

To run tests just execute mvn clean test - and it will compile the classes & execute tests for the modules.

Pipelines

MPL provided pipelines should be really simple, but could be improved with new best practices - so changes are always welcome.

Modules

If you have some interesting module - you for sure can prepare changes for existing module or a new module, write tests, create the pull-request, describe it - and after that this module could be approved to be included n the base library. If not - you always could use (or create) your own nested library to share this fancy module across your projects.