Open andrewhertog opened 6 years ago
[UPDATE] I think the issue is just that prometheus-operator and prometheus are sharing the same RBAC here. While they should be different. The other Cluster Role that should be used is: https://github.com/coreos/prometheus-operator/blob/v0.14.1/Documentation/rbac.md#prometheus-rbac and should be used as a different SA here https://github.com/aws-samples/aws-workshop-for-kubernetes/blob/master/02-path-working-with-clusters/201-cluster-monitoring/templates/prometheus/prometheus.yaml#L228
I realized that as the API Servers appeared to be down (which happened because the get
of /metrics
is not listed in the prometheus-operator cluster-role).
So the underlying problem is that there is a missing cluster role.
I also hit this issue - You can just kubectl edit clusterrole prometheus-operator -n monitoring
and add the missing verbs, in your case list
for endpoints/svcs and watch
for pods I think.
This is the RBAC that got the UI up in my case.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
[...]
rules:
- apiGroups:
- extensions
resources:
- thirdpartyresources
verbs:
- '*'
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- '*'
- apiGroups:
- monitoring.coreos.com
resources:
- alertmanagers
- prometheuses
- servicemonitors
verbs:
- '*'
- apiGroups:
- apps
resources:
- statefulsets
verbs:
- '*'
- apiGroups:
- ""
resources:
- configmaps
- secrets
verbs:
- '*'
- apiGroups:
- ""
resources:
- pods
verbs:
- watch
- list
- delete
- apiGroups:
- ""
resources:
- services
- endpoints
verbs:
- get
- list
- create
- watch
- update
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- namespaces
verbs:
- list
Interestingly, these are not required according to the doc of 0.14.1
Can someone make the changes that @CharlyF mentioned? As of 7/14/18 this was still not working.
I had the same issue and had to modify the cluster role.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus-operator
namespace: monitoring
rules:
- apiGroups:
- extensions
resources:
- thirdpartyresources
verbs:
- "*"
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- "*"
- apiGroups:
- monitoring.coreos.com
resources:
- alertmanagers
- prometheuses
- servicemonitors
- prometheusrules
verbs:
- "*"
- apiGroups:
- apps
resources:
- statefulsets
verbs: ["*"]
- apiGroups: [""]
resources:
- configmaps
- secrets
verbs: ["*"]
- apiGroups: [""]
resources:
- pods
verbs: ["list", "delete", "watch"]
- apiGroups: [""]
resources:
- services
- endpoints
verbs: ["get", "create", "update", "watch", "list"]
- apiGroups: [""]
resources:
- nodes
verbs: ["list", "watch"]
- nonResourceURLs:
- /metrics
verbs: ["get"]
- apiGroups: [""]
resources:
- namespaces
verbs: ["list"]
And to be a little more precise—if you followed the directions in the guide, and you have a blank Targets page (and the prometheus container in the prometheus-prometheus-1
pod is showing errors in the log like the ones shown earlier in this thread), then you need to:
prometheus-bundle.yaml
file directly.kubectl apply -f templates/prometheus/prometheus-bundle.yaml
again, to apply the changes.After a couple minutes, you should start seeing Targets 'UP' in the Prometheus UI.
I'll file a PR with this change, hopefully it can get merged soon!
I'm currently following https://github.com/aws-samples/aws-workshop-for-kubernetes/tree/master/02-path-working-with-clusters/201-cluster-monitoring
I've successfully loaded Prometheus in a browser after using the proxy command
kubectl port-forward $(kubectl get po -l prometheus=prometheus -n monitoring -o jsonpath={.items[0].metadata.name}) 9090 -n monitoring
but i am not seeing any of the metrics onlocalhost:9090
This is all I see:
I have gone through 201 from the beginning twice, with the same results, following the cleanup shown at the end of the tutorial
Update
I just did some digging and noticed a lot of the following in the logs for the prometheus-prometheus-0 pod: