Refactor and optimize the CI pipeline

[purelind]

Before asking a question, make sure you have

• [X] Searched existing Stack Overflow questions
• [X] Googled your question
• [X] Searched open and closed GitHub issues
• [X] Read the documentation

What is your question?

The current CI pipeline testing logic and environment preparation logic are mixed in the Groovy file, which makes it too complicated and incurs a high cost for understanding and modification.

The plan is to restructure CI Pipelines through intentional redesign, making the structure of the pipelins simpler and its execution more stable.

• Refactor the script-based Jenkins Groovy to declarative Groovy.
• The environment preparation script(like download third bin) is implemented using shell script instead of Groovy script.
• Using Prow to take over the triggering of pipelines.
• Implement the run group test script to support batch execution of tests in parallel Pods.

Which CI piplines will be affected?

• https://ci2.pingcap.net/view/ghpr_cdc/job/cdc_ghpr_integration_test/
• https://ci2.pingcap.net/view/ghpr_cdc/job/cdc_ghpr_kafka_integration_test/
• https://ci2.pingcap.net/view/ghpr_cdc/job/dm_ghpr_compatibility_test/
• https://ci2.pingcap.net/view/ghpr_cdc/job/dm_ghpr_integration_test/
• https://ci2.pingcap.net/view/ghpr_cdc/job/engine_ghpr_integration_test/

New CI pipelines

• https://do.pingcap.net/jenkins/job/pingcap/job/tiflow/

How to go about modifying the new pipelines

The definition of the assembly line is now simpler, optimized from over 1000 lines to around 150. https://github.com/PingCAP-QE/ci/tree/main/pipelines/pingcap/tiflow/latest

Which branches' pipelines will be affected?

This optimization only targets the pipeline of the master branch.

Which behaviors will change

1. The CI trigger command will change /run-integration-test --> /test cdc-integration-mysql-test /run-kafka-integration-test --> /test cdc-integration-kafka-test /run-dm-compatibility-tests --> /test dm-compatibility-test /run-dm-integration-tests --> /test dm-integration-test /run-engine-integration-test --> /test engine-integration-test /run-verify --> /test verify /run-all-tests --> /test all

2. CI status will change idc-jenkins-ci-ticdc/kafka-integration-test --> pull-cdc-integration-kafka-test idc-jenkins-ci-ticdc/integration-test --> pull-cdc-integration-mysql-test idc-jenkins-ci-ticdc/dm-compatibility-test --> pull-dm-compatibility-test idc-jenkins-ci-ticdc/dm-integration-test --> pull-dm-integration-test tiflow/engine-integration-test --> pull-engine-integration-test

How to check my pull requst ci status on prow

https://prow.tidb.net/?repo=pingcap%2Ftiflow&type=presubmit

Any others changes I need to pay attention to?

1. This refactoring optimization only targets the master branch, and other branches are not affected.
2. The PR batch merge will be supported in the future.

[purelind]

To unify the experience across all branches, the following active release branch pipelines will also be restructured and adjusted.

• [x] release-7.1
• [x] release-6.5
• [x] release-6.1
• [x] release-5.4
• [x] release-5.3