search

opendatahub-io / modelmesh-serving

Controller for ModelMesh
Apache License 2.0
3 stars 31 forks source link

Modelmesh Prometheus monitoring does not work in RHODS 2.4 #252

Open lugi0 opened 10 months ago

lugi0 commented 10 months ago

Describe the bug

The Prometheus instance of modelmesh monitoring does not work in RHODS 2.4 - there are no targets present and no queries can be run against it

To Reproduce Steps to reproduce the behavior:

  1. Go to Networking -> Routes in NS redhat-ods-monitoring
  2. Click on Prometheus Route
  3. Check targets in Status -> Targets or try to run a query
  4. No results / no targets are present

Expected behavior

Targets can be found and queries can be run

Screenshots image image

Environment (please complete the following information):

RHODS 2.4

Additional context

Metrics can be run in Openshift's Metrics page, they are present and working correctly. I could not find the expected targets anywhere else though.

zdtsw commented 10 months ago

what i can see from cluster: https://github.com/red-hat-data-services/odh-manifests/blob/rhods-2.4/modelmesh-monitoring/base/prometheus/rhods-prometheus.yaml#L36 is trying to use servicemonitor "modelmesh-serving" which uses service "modelmesh-metrics-monitor" to select pod with label "modelmesh-service=modelmesh-serving" but there is no pod with this lable in application namespace.