[component pipeline] Allow to deploy to multiple clusters

[michaelsauter]

Is your feature request related to a problem? It would be nice to deploy to multiple clusters (and within each cluster to multiple namespaces). This is a typical scenario, and currently users have to build it themselves.

Describe the solution you'd like odsComponentStageRolloutOpenShiftDeployment should be able to deploy to multiple clusters/namespaces. For this, it first needs to change from rolling out one deployment (identified by resourceName) to rolling out all deployments for the selector. Then, it needs to be able to read "environment configuration", that specifies for each environment the clusters / namespaces to deploy to. The configuration could/should be largely identical to https://www.opendevstack.org/ods-documentation/opendevstack/3.x/jenkins-shared-library/orchestration-pipeline.html#_environments, but it should be able to specify multiple clusters. Further, the configuration cannot be sourced from a repo/file, but must either be inline in the Jenkinsfile (which works well for one component) or it could be sourced from a ConfigMap in the *-cd project (which would be the better fit for multiple components with same/similar configuration).

Describe alternatives you've considered The orchestration pipeline is also missing this functionality. One could add the functionality there, and require users wanting to deploy to multiple clusters to use the orchestration pipeline. I would like to avoid this because:

• the orchestration pipeline is complex and imposes a certain process that is not required for "just" deploying to multiple clusters
• the orchestration pipeline bundles all components together, and there are good organisational and technical reasons to deploy individually (e.g. less risk, higher speed)

Additional context We also need to take care of importing images to the target clusters.

[metmajer]

Related to https://github.com/opendevstack/ods-jenkins-shared-library/issues/389

[metmajer]

As discussed with @michaelsauter, the following topics need to be addressed:

• D/P + P vs. D + Q/P: Will deployments into different P-environments differ and, if yes, in which ways? Do we need a Q environment next to each P, or is the setup (D/Q + P clusters) sufficient? Think of requirements depending on the geographic location (language, location-specific features, etc.)
• Atomicity: Do we require rollbacks if the application fails to deploy on a single cluster? Can we avoid rollbacks by implementing idempotence?
• TIP/TIR documents: Assumption: TIR for P to contain the P-cluster locations, but there's still only a single document generated for P.
• Selective Rollouts: what if there's a need to deploy selectively into a geographic region? Could the Release Manager solve this by supporting multiple branches with different dev/qa/prod.env files, where master contains a setup for all clusters (default)? How can we deploy the same release into another region at a later point in time? Can we ignite a release into P without going through D/Q again? Assumption: D/Q success -> P-NORAM success .... > P-APAC ...?
• Testing: How can we test all of this automatically with at least 2 clusters? Can we use ODS in a Box in such a test scenario?