[EPIC] Only run relevant CI for each change

[cmur2]

Introduction

Currently, the monorepo includes the CI pipelines of Zeebe, Operate and Tasklist side-by-side. They are not integrated and as noted in #17231 the path filters are lengthy, redundantly defined and not applicable to the merge queue. In the end we run CI pipelines of Zeebe, Operate and Tasklist more often than we need and also on unrelated changes, partly because there is no single entry point so using GitHub required status checks makes this duplication necessary. This has a few downsides:

• higher cost (large parts of the CI run on infrastructure that Camunda pays by use)
• less stability due to inherent flakiness (running unrelated flaky tests may fail the whole pipeline)
• confusing UI with 20+ very detailed and partially redundant status checks
• cumbersome configuration of GitHub required status checks

We can get some inspiration from the Console monorepo workflows where Clément recently established a more efficient and modular CI pipeline structure using change detection and conditional execution.

The scope of this Epic is to provide a foundation for unifying the existing CI pipelines running on code changes to have a single entry point and only running pipelines relevant for a change. More details in the planning doc.

Out of scope: scheduled/nightly pipelines, release workflows

Task Breakdown

✔️ Phase 1:

• ✔️ POC to adopt the examples from Console monorepo to camunda/camunda
  • goal: have a minimal example of how the change detection can work in context of camunda/camunda
  • must support push, pull_request and merge_group triggers
  • note: for merge queue integration see https://github.com/dorny/paths-filter/issues/183
  • results:
    • https://github.com/camunda/zeebe/pull/17826 + https://github.com/camunda/c8as-poc-unified-ci show that the mechanism can be adopted well for camunda/camunda
    • open questions: how to organize pipeline code for the monorepo, how to roll the change out

✔️ Phase 2:

• @cmur2 implements first iteration(s) as foundation
  • https://github.com/camunda/camunda/pull/19423

✔️ Phase 3:

• create staged implementation & rollout plan for camunda/camunda
  • note: iterative approach to deliver value early, aim for non-disruptive smooth rollout (old+new in parallel)
  • https://github.com/camunda/camunda/issues/21436
  • https://github.com/camunda/camunda/issues/21437
• get buy-in from stakeholders for plans
  • announcement

🚧 Phase 4:

• Stakeholders implement & migrate remaining functionality, e.g.
  • https://github.com/camunda/camunda/pull/19334
  • https://github.com/camunda/camunda/pull/19436
  • https://github.com/camunda/camunda/pull/19623
  • https://github.com/camunda/camunda/pull/19778
  • https://github.com/camunda/camunda/pull/19785
  • https://github.com/camunda/camunda/issues/18600
  • https://github.com/camunda/camunda/issues/21142

[Zelldon]

I created a follow-up https://github.com/camunda/camunda/issues/19539 to further investigate and improve related to further improve when to run which tests etc.