[issue-464]Create a Prometheus ServiceMonitor object that can capture/collect metrics from deployed SonataFlow instances

[jianrongzhang89]

Fix https://github.com/apache/incubator-kie-kogito-serverless-operator/issues/464.

The operator an track in the cluster if Prometheus Operator is installed. If this condition is true and the SonataFlowPlatform is marked with .spec.monitoring.enabled: true, and the workflow is deployed as an k8s deployment, then each workflow can have a ServiceMonitor object bound to it.

Note: for workflows deployed as Knative services, this PR does not create the ServiceMonitor objects.

Checklist

[X ] Add or Modify a unit test for your change [X ] Have you verified that tall the tests are passing?

[ricardozanini]

Please add conditionals when registering the listeners in the controller:

2024-10-01T14:32:35.471896952Z stderr F E1001 14:32:35.471807       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:32:45.472140785Z stderr F E1001 14:32:45.472033       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:32:55.471966197Z stderr F E1001 14:32:55.471868       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:33:05.472046816Z stderr F E1001 14:33:05.471956       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:33:15.472269129Z stderr F E1001 14:33:15.472185       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:33:25.472286385Z stderr F E1001 14:33:25.472204       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:33:35.4720388Z stderr F E1001 14:33:35.471924       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:33:45.472208024Z stderr F E1001 14:33:45.472121       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:33:55.472168441Z stderr F E1001 14:33:55.472041       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"
2024-10-01T14:34:05.471752553Z stderr F E1001 14:34:05.471677       1 kind.go:71] "msg"="if kind is a CRD, it should be installed before calling Start" "error"="no matches for kind \"ServiceMonitor\" in version \"monitoring.coreos.com/v1\"" "kind"={"Group":"monitoring.coreos.com","Kind":"ServiceMonitor"} "logger"="sonataflow-manager.controller-runtime.source.EventHandler"

[ricardozanini]

The e2e tests have been fixed, please rebase your branch 🙏

[jianrongzhang89]

Please add conditionals when registering the listeners in the controller:

Done!

[jianrongzhang89]

The e2e tests have been fixed, please rebase your branch 🙏

Code has been rebased.

[ricardozanini]

Can you check the files generation? 😬

[ricardozanini]

@jianrongzhang89 can you please rebase? Your Knative fix has been merged with the e2e refactoring.

[wmedvede]

@ricardozanini @jianrongzhang89

My last comment on this PR guys. Feel free to open a follow up issue if you believe makes sense.

case1: I have brand new OpenShift Local installation. I never installed the Prometheus operator I create a SFP with spec.monitoring.enabled = true I create a WF in namespace test After the WF deployment I can see a ServiceMonitor monitor created for it, weird. Only explanation is that OpenShift Local already has the operator installed the Prometheus resources ouf-of-the-box

In customer OpenShfift cluster installations different thant OpenShift Local installations IDK what will happen. Probably the operator is always there.

case2: I install the Prometheus operator in namespace test1 I create a SFP, with spec.monitoring.enabled = true I create, a WF, etc, I can see the ServiceMonitior, etc.

I create a namespace test2, I don't install the Prometheus operator in that namespace. I create a SFP, with spec.monitoring.enabled = true I create, a WF, etc, I can see that a ServiceMonitior, etc.

Sort of corolary o case1

[ricardozanini]

@wmedvede I think we can safely create the monitor always if the monitoring: true is enabled and Prometheus is available.

My only take is that we should report in events that monitoring: true and Prometheus is not available. This must be done in this PR. @jianrongzhang89 can you confirm?

[jianrongzhang89]

@wmedvede I think we can safely create the monitor always if the monitoring: true is enabled and Prometheus is available.

My only take is that we should report in events that monitoring: true and Prometheus is not available. This must be done in this PR. @jianrongzhang89 can you confirm?

@ricardozanini yes, in this case, a message will be logged in the platform log: I1018 15:05:40.135463 1 sonataflowplatform_controller.go:146] Monitoring is enabled in platform sonataflow-platform, but Prometheus is not installed

[ricardozanini]

@jianrongzhang89 Can you change to an event instead too?

[jianrongzhang89]

@jianrongzhang89 Can you change to an event instead too?

Done