Observe and Prometheus/Alertmanager API queries broken after upgrade 4.12 -> 4.16

[cannibalisticcow]

Describe the bug A user that has access to view the Observe panel and make queries to the endpoints; /api/prometheus/api/v1/rules and /api/alertmanager-user-workload/api/v2/alerts

On version 4.12 can no longer do this when the cluster is upgraded to 4.15. The Observe panel shows Forbidden and when looking at the network response the problem seems to be; Forbidden (user=robot, verb=get, resource=prometheuses, subresource=api) When querying the user-workload endpoint the problem is Forbidden (user=robot, verb=get, resource=alertmanagers, subresource=api)

I've been able to work around the problem by creating a new cluserrole and clusterrolebinding but feel like as this worked previously it should continue to work post upgrade. The user has a number of default custom roles applied to allow cluster metrics and state to be viewed. Default Roles applied

rules:
  - verbs:
      - get
    apiGroups:
      - ''
    resources:
      - namespaces
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - config.openshift.io
    resources:
      - clusterversions
      - infrastructures
  - verbs:
      - watch
      - list
    apiGroups:
      - ''
    resources:
      - events

rules:
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ''
    resources:
      - nodes
  - verbs:
      - list
      - get
      - watch
    apiGroups:
      - metrics.k8s.io
    resources:
      - nodes
      - pods
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - ''
    resources:
      - namespaces
  - verbs:
      - list
    apiGroups:
      - ''
    resources:
      - pods
      - services
  - verbs:
      - list
    apiGroups:
      - monitoring.coreos.com
    resources:
      - servicemonitors
      - podmonitors
  - verbs:
      - get
      - list
      - watch
    apiGroups:
      - config.openshift.io
    resources:
      - clusterversions
      - clusteroperators
      - infrastructures
  - verbs:
      - watch
    apiGroups:
      - events.k8s.io
    resources:
      - events
  - verbs:
      - list
    apiGroups:
      - machineconfiguration.openshift.io
      - machine.openshift.io
      - autoscaling.openshift.io
    resources:
      - machineconfigpools
      - machines
      - machinesets
      - machineautoscalers
      - machinehealthchecks
      - machineconfigs
  - verbs:
      - watch
      - list
    apiGroups:
      - ''
    resources:
      - events
      - persistentvolumeclaims
  - verbs:
      - watch
      - list
    apiGroups:
      - network.openshift.io
    resources:
      - netnamespaces
  - verbs:
      - watch
      - list
    apiGroups:
      - operators.coreos.com
    resources:
      - clusterserviceversions

Work around

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: monitoring-api-fix
rules:
  - verbs:
      - get
    apiGroups:
      - monitoring.coreos.com
    resources:
      - prometheuses/api
      - alertmanagers/api
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: monitoring-api-fix
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: 'system:authenticated'
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: monitoring-api-fix

Version 4.15.0-0.okd-2024-03-10-010116

How reproducible We have multiple OKD clusters, all of the ones that have been upgraded from 4.12 to 4.15 are displaying this problem. The ones that are still at 4.12 continue to work. I've not had the opportunity yet to see if this is a problem that was introduced in .13, .14, or .15

[cannibalisticcow]

Observe page functions normally in 4.13.0-0.okd-2023-09-30-084937

[cannibalisticcow]

Tested on 4.14.0-0.okd-2024-01-06-084517 Other than a bug with the plug-in rbac is working correctly for accessing the Observe page and querying the above endpoints.

[cannibalisticcow]

Breaks from 4.15.0-0.okd-2024-02-10-035534

[cannibalisticcow]

A 4.16 cluster has now been built too and the problem also exists there. 4.16.0-0.okd-scos-2024-08-14-110444

[JaimeMagiera]

Thanks for testing. We'll take a look.