feat(backend): Add Semaphore and Mutex fields to Workflow CR

[DharmitD]

Resolves https://github.com/kubeflow/pipelines/issues/6553

Description of your changes: This PR introduces support for Pipeline-level Semaphores and Mutexes in the KFP backend.

Changes Introduced:

• Added the ability to specify a semaphore for pipelines, which controls the number of concurrent instances of a pipeline that can run. The semaphore is configured via a fixed ConfigMap named semaphore-config. The semaphore key is provided through the pipeline configuration.

• Added mutex support for pipelines, ensuring that only one instance of the pipeline can run at a time if the specified mutex is defined. Mutex names are defined per pipeline, and each pipeline instance respects the specified mutex.

• The Workflow CR now includes a Synchronization field, where semaphore and mutex are appropriately set.

• If a pipeline has a semaphore, the backend maps the semaphore to the semaphore-config ConfigMap using the key provided by the user. Mutexes are represented by their name, ensuring mutual exclusion.

This PR should be merged only after https://github.com/kubeflow/pipelines/pull/11340 gets merged. Testing instructions

• Build the API Server image and push to an image registry

• Upload main.yaml file from here

• Check in KFP UI Pipeline Spec tab if the following snippet is present:

  platforms:
  kubernetes:
  pipelineConfig:
    mutexName: mutex
    semaphoreKey: semaphore

• After the pipeline run is initiated, use the following command to verify that the Workflow CR has the appropriate synchronization settings:

  oc get workflow -o yaml $(oc get workflow --no-headers | awk '{print $1}') | yq .spec.synchronization

• The expected output should include the semaphore and mutex references:

  synchronization:
  mutex:
    name: mutex
  semaphore:
    configMapKeyRef:
      key: semaphore
      name: semaphore-config

  Checklist:

• [x] You have signed off your commits

• [x] The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign chensun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[backend/OWNERS](https://github.com/kubeflow/pipelines/blob/master/backend/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[gregsheremeta]

add fixes #6553 to the PR description

[gregsheremeta]

The semaphore is configured via a fixed ConfigMap named semaphore-config

We should edit the kubeflow manifest to deploy a skeleton of this configmap. You can do that in here or in a follow-up PR.

[gregsheremeta]

The Workflow CR now includes a Synchronization field

I would probably delete this line (and maybe edit the PR title), because that reads like things you enhanced on Workflow itself. We're just setting fields on it...

[gregsheremeta]

platforms:
  kubernetes:
    pipelineConfig:
      mutexName: mutex
      semaphoreKey: semaphore

The expected output should include the semaphore and mutex references:

What does Argo Workflows do when both are set?

A better verification would be to do two separate test pipelines -- one where you use mutex, and one where you use semaphore. And then in addition to verifying the Workflow yaml, also verify that multiple runs are being locked like they should be.

[DharmitD]

/hold until https://github.com/kubeflow/pipelines/pull/11384 and https://github.com/kubeflow/pipelines/pull/11340 get merged