search

dmwm / CRABServer

15 stars 37 forks source link

Release pipeline #8421

Closed novicecpp closed 1 month ago

novicecpp commented 1 month ago

Complete the https://github.com/dmwm/CRABServer/issues/8401

The release pipeline is based on build-deploy-test pipeline but has 2 more stages:

The git tag name must match v3\.[0-9]{6}.* regex in order to trigger above jobs. The final docker image tag, RELEASE_NAME, will be v3.YYMMDD-stable as we agreed in the issue.

This change also allow to do force release by pushing the git tag name same as above condition (the convention is v3.YYMMDD__somestring), provides RELEASE_NAME=final-tag-name-including-stable and SKIP_DEPLOY=t variable to CI.

cmsdmwmbot commented 1 month ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/1993/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 1 month ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/1994/artifact/artifacts/PullRequestReport.html

belforte commented 1 month ago

I tried to look. But can't understand what the code does. Nor can I really understand the instructions. This is possibly too terse

This change also allow to do force release by pushing the git tag name same as above condition (the convention is v3.YYMMDD__somestring), provides RELEASE_NAME=final-tag-name-including-stable and SKIP_DEPLOY=t variable to CI.

novicecpp commented 1 month ago

This will be long explanation. Please bear with me.

2 things we rely on:

  1. We use the pushing tag (git push <remote> <tag>) as the way to trigger the pipeline.
  2. The release flow will be the same as we do now; go to the release page in GH and create a new release. From there, we use the tools to sync the release tag to Gitlab and trigger Gitlab CI.

(The second mechanism is still not yet finalized, but we can simulate it by simply pushing the tag with the same name as the release tag, directly to Gitlab to trigger the pipeline.)

"To trigger the pipeline" in this context means CI selecting the jobs in the .gitlab-ci.yml where the rules directive of the jobs are satisfied. Then, puts these jobs together and creates a pipeline to run. The rules for if directive is test against the CI variables defined each trigger. If the rule does not match, next rule will be evaluate. If no job matches the conditions, no pipeline will be run.

Reading the rule is a bit tricky here because Gitlab offered the yaml tag !reference as the way to reference the rule defined in hidden keys.

For example, job get_env at L26-L27 define the rule as:

get_env:
  rules:
    - !reference [.default_rules, default]

And the key .default_rules we define in here is:

.default_rules:
  default:
    - if: $CI_COMMIT_TAG =~ /pypi-.*/
    - if: $CI_COMMIT_TAG =~ /v3\.[0-9]{6}.*/

Make get_env equivalent to:

get_env:
  rules:
    - if: $CI_COMMIT_TAG =~ /pypi-.*/         
    - if: $CI_COMMIT_TAG =~ /v3\.[0-9]{6}.*/

This rule means to run this job only when $CI_COMMIT_TAG matches the regex pypi-.*, OR v3\.[0-9]{6}.*.

For $CI_COMMIT_TAG, this variable is only available when pushing the tag.

Here is the base "Build-Deploy-Test" pipeline which use the .default_rules:default:

320979971-67321507-67ff-4d62-98a6-0ba484b21d3a

Here is the "Release Pipeline" of this PR which some jobs only have .default_rules:release rules (#7420156):

Screenshot from 2024-05-22 11-29-00

In some cases, you want to run only some jobs, not all-or-none. One way you can do this is to let users provide some variables and add rules to explicitly not trigger the job. In our case, we want to skip the testing part to release the new image immediately.

For example, deploy_server job:

deploy_server:
  rules:
    - if: $SKIP_DEPLOY
      when: never
    - !reference [.default_rules, rules]

Which mean, if $SKIP_DEPLOY is exist, do not run the deploy_server. If not, checking next rule.

Here is the pipeline when $SKIP_DEPLOY is defined (#7401218):

Screenshot from 2024-05-22 13-24-32

You can supply any variable to the pipeline each for trigger, by providing the push option ci.variables=VARIABLE_NAME=value when pushing the tag.

The $RELEASE_NAME is generated from parseEnv.sh in the get_env job which simply appends the -stage to $CI_COMMIT_TAG, and will be used to re-tag docker image later in the release_stable job. However, to trigger the pipeline from the same commit again, you need the different git tag. Different tag mean different release image tag too. So, $RELEASE_NAME=tagname allows you to override the final image tag to whatever you want.

For example, push the tag with name v3.YYMMDD__manual1, and push variable $RELEASE_NAME=v3.YYMMDD-stable to have final docker image tag v3.YYMMDD-stable, and SKIP_DEPLOY=t to skip all testing steps.

(The code need some updates to make it more clear, will do it tomorrow).

belforte commented 1 month ago

hmm.. something like this ?

cd to my R/W clone of .... what's the gitlab repo name ?
(git clone ssh://git@gitlab.cern.ch:7999/crab3/CRABServer.git  ? )
git checkout master
git tag v3.240601__manual1
git push -o RELEASE_NAME=v3.240601-stable -o SKIP_DEPLOY=t

forgive me, I never used git to create tags, always did it via the web i/i in github, i.e. direction on upstream.

I am sorry but I can't possibly try to review and understand the code. I am sticking with "have instructions which I can follow" and then will test those.

novicecpp commented 1 month ago

Ok.

And Sorry for make explanation too long. The correct commands should be:

cd path/to/your/usual/forked/CRABServer
git remote add gitlab ssh://git@gitlab.cern.ch:7999/crab3/CRABServer.git 
git checkout master # make sure you build the release from master
git tag v3.240601__manual1
git push gitlab v3.240601__manual1 -o ci.variable="RELEASE_NAME=v3.240601-stable" -o ci.variable="SKIP_DEPLOY=t"

EDITED: fix git push command.

novicecpp commented 1 month ago

Hmm, look like it is easily to make a mistake when you forget to checkout master/v3.YYMMDD before tag.

Maybe trigger pipeline via Gitlab API is better, so you do not need to re-tag it. It is a bit inconvenient because you need separate credential (token API) in your machine instead using ssh/git mechanism to authenticate.

I will have a look later.

cmsdmwmbot commented 1 month ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/1996/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 1 month ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-CRABServer-PR-test/1997/artifact/artifacts/PullRequestReport.html

belforte commented 1 month ago

Do we need a client-side API with tokens etc ? Or could this be done on the gitlab web ui ?

Maybe it should be a multistep action ?

  1. tag and push v3.240601__manual1
  2. wait for build and validation
  3. push release name

btw, what's the bad side of forgetting to tag locally before pushing ? is that something that can be (easily) corrected in case ?

novicecpp commented 1 month ago

btw, what's the bad side of forgetting to tag locally before pushing ? is that something that can be (easily) corrected in case ?

Let imagine when you are in hurry and you want to release master immediately. You go to your local CRABServer directory, current branch is not master because you are testing new multiprocess feature, let call my-multiprocessor, then you tag, and you push to gitlab. The tag you push is actually from latest commit of your branch my-multiprocessor, so pipeline build your commit instead of from master.

This kind of situation I would like to avoid.

For how we trigger the pipeline to force release, push options is how I "hack" to force pipeline to run. I would like to start from "what should it be" first. But let talk about this in the issue.

belforte commented 1 month ago

thanks, now I understand

Hmm, look like it is easily to make a mistake when you forget to checkout master/v3.YYMMDD before tag.

and I agree with the concern