Problem: Hare pull requests are not tested by CI

[vvv]

Solution: enable a Jenkins pipeline that gets triggered whenever new pull request is created or its branch is updated. This pipeline must support PRs created from forks of https://github.com/Seagate/cortx-hare.

To Do

• [x] pipeline starts automatically when new PR is created or its branch is updated
  • See GitHub hook trigger for GITScm polling.
• [x] concurrent CI job don't compete over the VM
  • ‘Lockable Resources Manager’ in Jenkins configuration
  • Pipeline Steps documentation
  • Lockable Resources plugin documentation
• [x] proper repository is git-cloned by the test (currently the URI is hard-coded in Jenkinsfile)
• [x] required motr and build dependencies are installed.
• [x] singlenode cluster is bootstrapped
• [ ] put VM_USER_PASS_CRED_ID into Jenkins' credentials (currently it's in Jenkinsfile as plain text)
• [x] RPM build and Unit tests based on RPM installation.
• [x] port test-boot-1
• [ ] [optional] port test-boot-1-s3 (if applicable)
• [ ] [optional] port test-boot-2 (if applicable)
• [ ] [optional] port test-boot-3 (if applicable)
• [ ] think how to provide/publish results of test run

“Map” of Hare CI

[vvv]

cc @azheregelya @shailesh-vaidya

[vvv]

I've created a Multibranch pipeline to experiment with — vvv.hare-multibranch.

[vvv]

From Jenkins' GitHub Hooks Problems page:

There are no credentials with admin access to manage hooks on GitHubRepositoryName[host=github.com,username=seagate,repository=cortx-hare]

[azheregelya]

I've checked Jenkins configuration.

• We have all required plugins installed. Good news.
• ~It seems that hooks are not managed. The checkbox "Manage hooks" is not set in Jenkins -> Configuration -> Github.~
• Moreover, it seems that there is a requirement for GitHub webhooks which might not be met yet:

  The Hook URL is http://eos-jenkins.colo.seagate.com/github-webhook/ , and it needs to be accessible from the internet.

[vvv]

I've created a Multibranch pipeline to experiment with — vvv.hare-multibranch.

OK, maybe Multibranch pipeline is not needed after all. According to Jenkins documentation,

The Multibranch Pipeline project type enables you to implement different Jenkinsfiles for different branches of the same project. In a Multibranch Pipeline project, Jenkins automatically discovers, manages and executes Pipelines for branches which contain a Jenkinsfile in source control.

We don't need different Jenkinsfiles for different branches... Or maybe we do... 🤔 I don't know for sure.

[vvv]

[ ] concurrent CI job don't compete over the VM

There is no need to add lockable resources to global Jenkins configuration.

From lock documentation:

• resource The resource name to lock as defined in Global settings. If the resource does not exist in Global Settings it will be automatically created on build execution.

From the plugin documentation:

When the lock step is used in a Pipeline, if the resource to be locked isn't already defined in the Jenkins global configuration, an ephemeral resource is used: These resources only exist as long as any running build is referencing them.

cc @azheregelya @shailesh-vaidya

[vvv]

• [ ] proper repository is git-cloned by the test (currently the URI is hard-coded in Jenkinsfile)

From CloudBees' Understanding and implementing Pipeline as Code:

Multibranch Pipeline projects expose the name of the branch being built with the BRANCH_NAME environment variable and provide a special checkout scm Pipeline command, which is guaranteed to check out the specific commit that the Jenkinsfile originated. If the Jenkinsfile needs to check out the repository for any reason, make sure to use checkout scm, as it also accounts for alternate origin repositories to handle things like pull requests.

[vvv]

• [x] pipeline starts automatically when new PR is created or its branch is updated

Jenkins server is behind VPN, GitHub cannot send webhook event to it. I've configured Jenkins pipeline to rescan the repository every minute.

[shailesh-vaidya]

@vvv Jenkins is not accessible over the internet. Hence we are not able to use GitHub Web-hook configuration to trigger builds. We are using polling mechanism (as you already configured) for our Jenkins jobs. We are using multi-branch pipelines for other component CI (pre-merge jobs) with polling.

I would request to use Dev Jenkins for any experiments. To avoid any issues on Production Jenkins.

[azheregelya]

@vvv I suspect that disableConcurrentBuilds() and lock('hare-ci-vm') do the same job for us though lock offers better granularity. https://www.jenkins.io/doc/book/pipeline/syntax/

[vvv]

@amolkongre wrote (in direct message):

Hi Valery ... just wanted to know if hare CI on Jenkins is working fine or not?

@amolkongre Short answer: not quite.

@azheregelya You've put quite some time into this task (thanks!). Would you describe the current plan of reenabling Hare CI and outline the remaining steps in the issue description? Thank you so much.

[azheregelya]

@vvv updated description @amolkongre Hare CI is not merged to dev branch yet. Major stages are implemented and some tests are already functioning. Currently adding more stages with more tests to reach similar functional test coverage as it was on Gitlab CI.

[azheregelya]

test-boot1 is ported.

[azheregelya]

Pull request is ready for review.