Split Metric and Lifecycle into two distinct operators

[thisthat]

Goal

Split the current operator into two separate operators, one for each CRD group.

Technical Details

There are two main reasons why this change is beneficial for end users:

• we can scale resources/replicas independently
• if we use only one feature and not the other

Furthermore, we can restrict the permissions for each operator with more fine granularity.

This split can cause issues during upgrades. In the case the new metric operator is up-and-running while the old single-operator runs, there are two controllers on the same resource that try to write. Bumping the Metric CRD version should suffice to avoid this problem.

General action plan

• split operators
• remove KeptnEvaluationProvider in favor of having a single provider that points to the metrics operator
• create new KeptnMetricsProvider CRD which references external o11y providers (prometheus, dynatrace,...) in the new metrics operator
• move KeptnMetrics to metrics operator
• have KLT request metric values from the KeptnMetrics CR

Acceptance Criteria

• [x] Common code between the operators is shared in a new module (e.g., Eval Provider)
• [x] Permissions are restricted to the minimum for each operator (e.g., https://github.com/keptn/lifecycle-toolkit/blob/main/operator/config/rbac/extra_roles.yaml)
• [x] Cleanup of the left-over APIs reference (e.g., PROJECT file)

DoD

• [x] Metric CRDs version is bumped to v1alpha2
• [x] Two operators are available
• [x] Tests are adapted

### Tasks
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/845
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/846
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/847
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/848
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/849
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/850
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/851
- [x] adapt docs
- [x] adapt examples
- [ ] ???
- [ ] profit
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/942
- [ ] https://github.com/keptn/lifecycle-toolkit/issues/960

[bacherfl]

Looking into this I found some questions that we need to address:

• With the KeptnEvaluationProvider CRD being used by both of the operators, we probably need to move this into the new metrics-operator as well to avoid cyclic dependencies between both operators (since the previous operator will also refer to KeptnMetric CRDs). The problem here is however that the KeptnEvaluationProvider CRD is part of the lifecycle api group, so we'd have part of this group in the new metrics-operator as well, which feels unclean. Probably we'll have to move this into a common package, but given that the EvaluationProvider structs represent a CRD that's set up with kubebuilder, we'll have to find a best practise for sharing CRDs between the two modules containing the operators using them

[mowies]

Sharing CRDs is generally not a best practise I think 😄 But didn't we plan to have some generic endpoint in the metrics operator to query metrics from? I think that would also be the place for KLT to fetch metrics from. And then the rest is handled in the metrics operator. That way we could achieve clear separation of concerns.

In any case I would suggest to keep API groups in one operator and not split them over 2 operators.

[thisthat]

If we can't easily have a shared "CRD", I agree with @mowies. The Providers shall be moved into the metric-operator. The lifecycle-operator should only use keptn-metric for evaluations. However, we have a shared read and not write of the KeptnEvaluationProvider

[bacherfl]

I think there was a plan for retrieving metrics from that generic endpoint. However, right now we also support KeptnEvaluationDefinitions referring to e.g. Prometheus and Dynatrace EvaluationProviders directly as well (i.e. not only the KeptnMetrics). So the problem is that in the metrics.KeptnMetrics we currently depend on lifecycle.KeptnEvaluationProvider, which is also a dependency of lifecycle.KeptnEvaluationDefinition.

In my opinion, the cleanest approach would be to introduce something like metrics.KeptnMetricsProvider that can be used by the metrics.KeptnMetrics controller, and only refer to metrics.KeptnMetrics in the lifecycle.KeptnEvaluations - here we could then also go via the generic endpoint, as @mowies suggested