Build RPM package for every created or updated PR

[TimothyAsirJeyasing]

(2018-02-07 updated/rewritten by @dahorak)

We would like to be able to automatically build RPM packages for every created or updated PR. Build status should be reported back to the respective PR. Packages should be available in some repository for any additional testing.

The intended workflow is:

• Every new or updated PR will trigger package build job in CentOS CI Jenkins.
• The job might be triggered also by comment run build in the respective PR.
• The job will create rpm repository with packages related to the particular git repo/PR. The repositories will be hosted on http://artifacts.ci.centos.org/.
• ❔ Prospectively some basic validation (unit tests execution, rpmlint) might be performed.

Functional testing of the PR builds is covered in another issue: https://github.com/Tendrl/tendrl-ci/issues/10.

[TimothyAsirJeyasing]

@mbukatov, @dahorak -> Please review/update the details. @r0h4n, @nthomas-redhat, @shtripat -> please review.

[shtripat]

@TimothyAsir do we need PR to be merged in master for feature build? I feel without merge we should be able to cherrypick the PR and do a build. What you think?

@r0h4n comments?

[dahorak]

@TimothyAsirJeyasing sorry for later response.

Did I understand it correctly, that we want to:

• monitor for any commit into PR in all the Tendrl repos ->
• trigger new build (in copr) from the (not yet merged) PR ->
• store the build in repo (specific for the particular PR) for further usage ->
• prepare testing cluster using the packages from the PR ->
• run some testsuite on the cluster and report results?

[TimothyAsirJeyasing]

@dahorak yes you are right, that's what its required.

[dahorak]

Triggering package build in CentOS CI for each pull request should be possible[1].

On the repository level we have to:

1. add the 'centos-ci' user as a collaborator (to each repository) !!This gives ci.centos.org commit access in order to set the test status on each pull request!!
2. add the Github Pull Request Builder webhook as described on the Wiki[1] (section Mode 2. Trigger a build for each Pull Request).

Everything else will be done in the job configuration (I'll create some proposal and send PR later).

[1] https://wiki.centos.org/QaWiki/CI/GithubIntegration

[nthomas-redhat]

@dahorak , That is a good start.

Also we need to think about:

• Where to keep the builds?
• How to prepare the test machines, install and trigger the tests
• How to handle the PR dependencies; i.e. a PR depends on another PR(otherwise tests might break)

[dahorak]

Where to keep the builds?

For now, I would keep the builds in copr/tendrl/Features or similar repo (and for the test execution on a temporary test server which will be part of the testing cluster created ad hoc in CentOS CI).

We can also consider usage of CentOS CI Artifacts.

How to prepare the test machines, install and trigger the tests

We already have one testing cluster automated in CentOS CI (by default for packages from release repo). The test machines are prepared using ansible playbooks [1, 2]. So it will be similar (with some additional updates to be able to consume the PR builds.

How to handle the PR dependencies; i.e. a PR depends on another PR(otherwise tests might break)

I have the same question and I'm not sure how to do that...

[dahorak]

@nthomas-redhat @TimothyAsirJeyasing the source for the PR build should be taken from refs/pull/XYZ/head or refs/pull/XYZ/merge (or both)? If I understand it correctly, the head points to the last commit on the "to be merged" branch, while the merge points to already merged version (how it will look after the merge).

[TimothyAsirJeyasing]

@nthomas-redhat Could you please add "centos-ci' as a user to all our tendrl repos so that the centos ci will get access to post the results and other stuff ?

This can be done by the following steps: In our every tendrl repository visit the Collaborators tab in the repo Settings Search by username 'centos-ci' and click 'Add Collaborator'

[dahorak]

@nthomas-redhat Can you add "centos-ci' as a user to all our tendrl repos so that the centos ci will get access to post the results and other stuff ?

For reference see CentOS CI Wiki - section Mode 2: Trigger a build for each Pull Request.

[nthomas-redhat]

@TimothyAsirJeyasing , @dahorak , Ack

[dahorak]

@nthomas-redhat Could you please also check, which Webhooks are configured in the repositories? Accordingly to the CentOS CI Wiki, there should be at least Pull Request and Issue Comment in the Let me select individual events section selected. If it is configured differently, could you please update it (for now at least in the api repo, which I'm using now for testing the jobs)?

[r0h4n]

@dahorak Who are the admin users from Tendrl github org that can trigger the tests automatically on centos ci? How are we automating that since it requires a comment on PR from whitelisted admin user everytime you want trigger the tests

[r0h4n]

I have added the required centos ci webhooks for PR builds in each repo

[dahorak]

@r0h4n Thanks for the webhooks!

The intended state is, that every PR from somebody from Tendrl organization will automatically trigger the build/tests. The comments (like run tests) are there just as additional possibility, for example to retest something or to trigger tests for PR submitted by somebody not in the Tendrl organization (this is because of security reasons).

[r0h4n]

Ok sounds good, I have also added "centos-ci" user to the Tendrl organization and to indvidual repos with "write" access as Collaborator.

Let me know if you need anything else

[dahorak]

Hopefully, there will be not much "spam" from the centos-ci while I'm debugging the jobs :-)...

[dahorak]

It will definitely need some more tweaking, but now we have one job for each Tendrl repository (CentOS CI tendrl-PR view). The particular job should be triggered by new or updated PR or comment [re]run tests (there might be required some more tweaking of the list of users whose update can trigger the build). The job will build snapshot package for the particular PR with added prXYZ to the version/release string (where XYZ is the PR number). The package is submitted to tendr/Features Copr repo. Additionally separated repo with just the packages related to the PR is created and uploaded to artifacts.ci.centos.org (for example: http://artifacts.ci.centos.org/tendrl/pr/tendrl-api/pr345.merge/).

Jobs are configured not to run more concurrent jobs (for one repo), so if there are multiple PR/updates in one repo at once, it should trigger multiple jobs, but they will be executed one by one consecutively (we might consider to allow them run concurrently though).

[TimothyAsirJeyasing]

There will be an individual jobs created for every tendrl repo to run the functional tests whenever a pr is submitted to a repo Ex. For review: https://ci.centos.org/view/tendrl-PR/job/tendrl-build-1-pr-api Each job does the following things:

• check if build_server is available
• prepare remote working directory
• clone git repository with source code
• fetch and checkout the PR branch
• build srpm package
• prepare copr configuration
• build package in copr
• check status of copr build
• download the packages from the copr
• prepare rpm repo
• prepare repofile
• upload generated repo to artifacts.ci.centos.org
• cleanup older packages for the particular PR
• request the machines
• install the packages and run the script

[dahorak]

Timothy basically summarize it (the last two steps are not implemented yet).

Here is PR with the job configuration: https://github.com/Tendrl/tendrl-ci/pull/8 (still WIP, so please do not merge).

About the missing steps (cluster installation):

• We already have long time working jobs for creating and installing cluster, presented by following two jobs: https://ci.centos.org/view/Tendrl/job/tendrl-2-0-cluster-gluster-create/ https://ci.centos.org/view/Tendrl/job/tendrl-2-1-cluster-gluster-install/ (there are additional jobs desired for basic testing, but those jobs are now working properly since last summer).
• The installation is by default based on packages from release repo, but is configurable. The problem is, that the installation playbooks (based on roles from tendrl-ansible) expect just two repos - one common for all Tendrl packages and one for dependencies. So we will have to update the setup playbooks to be able to supply multiple Tendrl (with packages from PR builds).

The question would be, if we wan't to have everything in one big job (build PR packages, validate the packages, prepare and install testing cluster,...), or if we want to split it to multiple jobs? Have everything in one big job will allow us easily track and report status for each PR. But it will take quite long time to run one job.

[dahorak]

@r0h4n there seems to be still some problems with updating the status back to github. Do you know, if it is required to somehow accept the invitation from the centos-ci site? I'm asking because I'm not sure, if the problem is with the job configuration of my job, or if it might be just unaccepted invitation from centos-ci site.

//update: I've send email to CentOS ci-users mailing list with question about this.

[dahorak]

Ok, the notification back to github seems to be fixed. The status is reported in the Checks section as CentOS CI results.

There seems to be still some remaining issue with whitelisted organization (members of Tendrl organization should be able to trigger the build, which I'm not sure if it works correctly).

[dahorak]

Accordingly what we discussed yesterday, my intention will be following functionality:

Every new or updated PR will trigger package build job

• The job might be triggered also by comment run build in the respective PR.
• The job create rpm repository with packages related to the particular git repo/PR.
• :grey_question: Prospectively some basic validation (unit tests execution, rpmlint) will be performed.

Once PR will be ready for merge, somebody will manually trigger test job

• The job will be triggered via comment run tests in the respective PR
• :grey_question: It might be possible to somehow specify dependencies on PRs from other repos (probably in the PR description or the run tests... comment).

[dahorak]

I'll keep this issue just for tasks related to building PR packages and I'll create new issue for executing tests suite.