search

carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
740 stars 200 forks source link

prometheus-k8s-0 stuck pending #3

Closed jmreicha closed 6 years ago

jmreicha commented 6 years ago

Thank you for the awesome project.

I ran the ./deploy script but just have one issue trying to bring everything up. The prometheus-k8s-0 and grafana pods seem to be stuck pending.

...
monitoring       grafana-68c8dcd4dd-bv7ng                0/1       Pending   0          17m
monitoring       prometheus-k8s-0                        0/2       Pending   0          15s
monitoring       prometheus-operator-c65785d89-2xkn4     1/1       Running   0          10m
...

relevant kubectl get events -n monitoring

5m          12m          26        prometheus-k8s-0.1534b4e7ea27ebb6                      Pod                                                            Warning   FailedScheduling        default-scheduler             pod has unbound PersistentVolumeClaims (repeated 4 times)
38s         3m           10        prometheus-k8s-0.1534b562df4c937a                      Pod                                                            Warning   FailedScheduling        default-scheduler             pod has unbound PersistentVolumeClaims (repeated 4 times)
1m          12m          43        prometheus-k8s-db-prometheus-k8s-0.1534b4e7214bad48    PersistentVolumeClaim                                          Normal    FailedBinding           persistentvolume-controller   no persistent volumes available for this claim and no storage class is set
12m         12m          1         prometheus-k8s.1534b4e721f847d6                        StatefulSet                                                    Normal    SuccessfulCreate        statefulset-controller        create Claim prometheus-k8s-db-prometheus-k8s-0 Pod prometheus-k8s-0 in StatefulSet prometheus-k8s success
12m         12m          9         prometheus-k8s.1534b4e72367ee8e                        StatefulSet                                                    Warning   FailedCreate            statefulset-controller        create Pod prometheus-k8s-0 in StatefulSet prometheus-k8s failed error: pods "prometheus-k8s-0" is forbidden: error looking up service account monitoring/prometheus-k8s: serviceaccount "prometheus-k8s" not found
3m          12m          2         prometheus-k8s.1534b4e7ea855e82                        StatefulSet                                                    Normal    SuccessfulCreate        statefulset-controller        create Pod prometheus-k8s-0 in StatefulSet prometheus-k8s successful
3m          3m           1         prometheus-k8s.1534b562e3e10eeb                        StatefulSet                                                    Warning   FailedCreate            statefulset-controller        create Pod prometheus-k8s-0 in StatefulSet prometheus-k8s failed error: The POST operation against Pod could not be completed at this time, please try again.

Any thoughts on what might be going on? I see that it wants a persistent volume, any way around that?

jmreicha commented 6 years ago

I just commented out/modified the references to persistent storage in manfiest/grafana-deployment.yml and manifest/prometheus/prometheus-k8s/prometheus-k8.yaml. Not ideal but works.

carlosedp commented 6 years ago

The deployments have persistent storage on a StorageClass that is default on my cluster. The claim is created by https://github.com/carlosedp/prometheus-operator-ARM/blob/master/manifests/grafana/grafana-claim.yaml that is mounted for each pod.

...
      - name: grafana-storage
        persistentVolumeClaim:
          claimName: grafana-claim

To keep persistent storage, you need to adjust these manifests for your system, using the volumes on a shared server (NFS, etc) or creating a StorageClass like I do here: https://github.com/carlosedp/kubernetes-arm/tree/master/3-NFS_Storage