ci: Refactor and optimize the CI pipelines

[purelind]

Enhancement

The current pipelines has some issues, which have brought a significant burden to the maintenance and updates of the pipelines.

• testing logic and environment preparation logic are mixed in the Groovy file
• There is a triggering relationship between the upstream and downstream calls of the pipelines.

The plan is to restructure CI Pipelines through intentional redesign, making the structure of the pipelins simpler and its execution more stable.

• Refactor the script-based Jenkins Groovy to declarative Groovy.
• The environment preparation script(like prepare cache and tools) is implemented using shell script instead of Groovy script.
• Using Prow to take over the triggering of pipelines.
• Remove the build-common CI pipeline and directly handle related tasks in the unit-test and integration-test pipelines.
• Each branch has an independent pipeline configuration to avoid adding conditional logic in the pipeline that is compatible with the differences of multiple branches.

Which CI piplines will be affected?

• https://ci.pingcap.net/view/tiflash/job/tiflash-build-common/
• https://ci.pingcap.net/view/tiflash/job/tiflash-ghpr-build/
• https://ci.pingcap.net/view/tiflash/job/tiflash-ghpr-integration-tests/
• https://ci.pingcap.net/view/tiflash/job/tiflash-ghpr-unit-tests/

New CI pipelines

• https://do.pingcap.net/jenkins/job/pingcap/job/tiflash/

Which branches' pipelines will be affected?

This optimization only targets the pipeline of the master branch. The remaining release branches will be switched and updated in the coming weeks.

Which behaviors will change

After this adjustment, the pipelines will be reduced from three to two. The functionality steps of tiflash_ghpr_build will be integrated into pull-integration-tests as a whole.

old pipelines

• tiflash-ghpr-unit-tests
• tiflash-ghpr-integration-tests
• tiflash-ghpr-build

new pipelines

• pull_unit_test
• pull_integration_test

1. The CI trigger command will change /run-unit-test --> /test pull-unit-test /run-integration-test --> /test pull-integration-test /run-all-tests --> /test all

2. CI status will change idc-jenkins-ci-tiflash/integration-test --> pull-integration-test idc-jenkins-ci-tiflash/unit-test --> pull-unit-test

3. Automatic triggering of the pielines For newly created PRs or PRs with new commits pushed, the pipeline will be triggered automatically. No need to comment /test all to trigger manually. Both pull-unit-test and pull-integration-test will be triggered automatically.

4. Prefer to use /retest to trigger tasks that have failed, /test all will retrigger all tasks (including those that have run successfully and those that have failed).

How to check my pull requst ci status on prow

https://prow.tidb.net/?repo=pingcap%2Ftiflash&type=presubmit

Any others changes I need to pay attention to?

• This refactoring optimization only targets the master branch, currently other branches are not affected.
• The PR batch merge will be supported in the future.

[purelind]

To unify the experience across all branches, the following active release branch pipelines will also be restructured and adjusted in the next few weeks

• [ ] release-8.1
• [ ] release-7.5
  • [ ] release-7.1
  • [ ] release-6.5
  • [ ] release-6.1

[purelind]

After several weeks of stability testing, the stability of the new pipeline on the master branch is observed to be consistent with the existing pipeline.

[JaySon-Huang]

Just a ref, the fixed of instability caused by printing the strerrno in new pipeline https://github.com/pingcap/tiflash/pull/8694