Make ODS pipeline configurable for MRO

[ungerts]

Multi repository orchestration (MRO) requires to checkout an ODS component/repository within a top level pipeline and to load/execute its Jenkinsfile dynamically. Example:

dir('componentId') {
   git url: '<REPO_URL>'     // checkout ODS component
   load path: 'Jenkinsfile'  // load and execute Jenkinsfile of ODS component
}

In order to allow orchestration the ODS pipeline should be more configurable:

• Activation/deactivation of CI Skip. In this case Skip is the task of the top level pipeline
• Activation/deactivation of Bitbucket notification. In case of an orchestrated build notifications are send by the top level pipeline.
• Activation/deactivation of setting build display name. Jenkinsfile runs within a top level Jenkinsfile and should not overwrite the build's display name.
• Activation/deactivation of repo checkout on Jenkins Master. Repo is already checked out.

PoC (working in progress) can be found here: https://github.com/ungerts/ods-jenkins-shared-library/tree/experimental/mro

[michaelsauter]

Plan sounds good 👍

[ungerts]

Implemented first version which allows me to run following Jenkinsfile for testing:

def final projectId = 'project13'
def final componentId = 'component123'
def final credentialsId = "${projectId}-cd-cd-user-with-password"
def sharedLibraryRepository
def dockerRegistry
def multiRepoBuild

node {
  sharedLibraryRepository = env.SHARED_LIBRARY_REPOSITORY
  dockerRegistry = env.DOCKER_REGISTRY
  multiRepoBuild = !!env.MULTI_REPO_BUILD
}

library identifier: 'ods-library@production', retriever: modernSCM(
        [$class: 'GitSCMSource',
         remote: sharedLibraryRepository,
         credentialsId: credentialsId])

def configMap = [
        image: "${dockerRegistry}/cd/jenkins-slave-golang",
        projectId: projectId,
        componentId: componentId,
        branchToEnvironmentMapping: [
                'master': 'test',
                '*': 'dev'
        ]
]

if (multiRepoBuild) {
  configMap.displayNameUpdateEnabled = false
  configMap.localCheckoutEnabled = false
  configMap.bitbucketNotificationEnabled = false
  configMap.ciSkipEnabled = false
  configMap.cloneSourceEnv = false
}

odsPipeline(configMap) { context ->
  //...
}

[ungerts]

Assumption ist that a MRO sets an environment variable called _MULTI_REPOBUILD

[michaelsauter]

Hmm - what do you think about hiding all of this inside the shared lib? You can read the environment variables inside (we only read SHARED_LIBRARY_REPOSITORY and DOCKER_REGISTRY because we need them before we have the shared lib), and then you can disable inside the lib. That way, we do not need to touch all the Jenkinsfiles out there and we do not need to have all this extra code in each file, making it hard to update.

[ungerts]

Yes, I think in that direction, too. My only counter argument is that in this case we access a "magic variable" (MULTI_REPO_BUILD) that does not exist in ODS at the moment.

Am Mi., 17. Juli 2019 um 17:19 Uhr schrieb Michael Sauter < notifications@github.com>:

Hmm - what do you think about hiding all of this inside the shared lib? You can read the environment variables inside (we only read SHARED_LIBRARY_REPOSITORY and DOCKER_REGISTRY because we need them before we have the shared lib), and then you can disable inside the lib. That way, we do not need to touch all the Jenkinsfiles out there and we do not need to have all this extra code in each file, making it hard to update.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/opendevstack/ods-jenkins-shared-library/issues/97?email_source=notifications&email_token=AALWPCXENP6OCRJWP2IDON3P742AHA5CNFSM4IEAG3J2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2EWQ2Q#issuecomment-512321642, or mute the thread https://github.com/notifications/unsubscribe-auth/AALWPCQSNZ6D6IHFQCRJVKTP742AHANCNFSM4IEAG3JQ .

[michaelsauter]

I'm okay with that. Maybe allow to also "enable MRO mode" by setting an option.

[ungerts]

@michaelsauter Found a solution that both is possible

[ungerts]

@michaelsauter @metmajer @tjaeschke how do we handle branch to environment mapping in MRO builds?

Options:

• branch to environment mapping is handled by the ODS branchToEnvironmentMapping mechanism and configured in the Jenkinsfile. I'm not sure whether this is a limitation, if using release branches. In this case the same branch is used in different environments (TEST/PROD).
• The orchestrating pipeline provides environment information using a variable which overwrites the mapping in the Jenkinsfile.

[clemensutschig]

+1 for option 2

[michaelsauter]

I'd also let the orchestrating pipeline control where things should be deployed.

[ungerts]

Implemented option 2

[ungerts]

Implemented approach:

Assumptions, Limitations, Deviations

• In case of an MRO build, the ODS Jenkinsfile must be called by the enclosing pipeline within a node.
  • Environment variable MULTI_REPO_BUILD must be set to true.
  • Environment variable MULTI_REPO_ENV must be set to the environment the pipline should deploy to (e.g. test).
• In case of an MRO build, no new node master is created by the ODS pipeline. The node of the enclosing pipeline is used, because creating a new node might change the working directory. Then the existing checkout cannot be used. Furthermore, the branch is not determined using the Openshift build config.
• On the agent checkout is done using the pipeline step git. The step checkout scm would checkout the repo of the enclosing pipeline.

MRO build configuration

• Environment is overwritten by variable MULTI_REPO_ENV.
• Clone environment is suppressed.
• Bitbucket notifications are disabled.
• Local checkout on Jenkins master is disabled.
• Display name of running build is not updated.
• CI skip is disabled

[michaelsauter]

If I understand correctly, this has been merged by now by another PR. Closing this - @ungerts reopen if you think this is not completed yet.