PrometheusOperatorRejectedResources

[goern]

Alertmanager URL: https://console-openshift-console.apps.nostromo.erdgeschoss.b4mad.emea.operate-first.cloud/monitoring

• firing https://console-openshift-console.apps.nostromo.erdgeschoss.b4mad.emea.operate-first.cloud/monitoring/graph?g0.expr=min_over_time%28prometheus_operator_managed_resources%7Bjob%3D%22prometheus-operator%22%2Cnamespace%3D~%22openshift-monitoring%7Copenshift-user-workload-monitoring%22%2Cstate%3D%22rejected%22%7D%5B5m%5D%29+%3E+0&g0.tab=1

  Labels:

  • alertname = PrometheusOperatorRejectedResources
  • container = kube-rbac-proxy
  • controller = prometheus
  • endpoint = https
  • environment = nostromo
  • instance = 10.128.0.194:8443
  • job = prometheus-operator
  • namespace = openshift-user-workload-monitoring
  • openshift_io_alert_source = platform
  • org = b4mad
  • pod = prometheus-operator-59cd6d448b-g7hgk
  • prometheus = openshift-monitoring/k8s
  • region = emea
  • resource = ServiceMonitor
  • service = prometheus-operator
  • severity = warning
  • state = rejected

  Annotations:

  • description = Prometheus operator in openshift-user-workload-monitoring namespace rejected 1 prometheus/ServiceMonitor resources.
  • summary = Resources rejected by Prometheus operator

TODO: add graph url from annotations.

[goern]

after looking through the logs of the prometheus-operator for user-workload-monitoring, for example https://console-openshift-console.apps.nostromo.erdgeschoss.b4mad.emea.operate-first.cloud/k8s/ns/openshift-user-workload-monitoring/pods/prometheus-operator-59cd6d448b-d8krg it showed that the openshift-operators namespace is missing the cluster-monitoring label. added it via oc label namespace openshift-operators openshift.io/cluster-monitoring=true.

https://github.com/operator-framework/operator-sdk/issues/4764 might be the root cause.

[goern]

same problem on phobos, here it is openshift-storage/lvms-operator-metrics-monitor

[goern]

https://access.redhat.com/solutions/6706741 was helpful