search

carlosedp / cluster-monitoring

Cluster monitoring stack for clusters based on Prometheus Operator
MIT License
740 stars 200 forks source link

Deployment failing: persistentvolumeclaim "grafana-storage" not found #71

Closed charlie-haley closed 4 years ago

charlie-haley commented 4 years ago

I'm getting a failed deployment with this error: persistentvolumeclaim "grafana-storage" not found. Any ideas on what could be causing this?

charlie-haley commented 4 years ago

I went to take a look at this again and it's now showing a different error, Grafana is actually loading now too.

running "VolumeBinding" filter plugin for pod "grafana-6ccccd4858-qrctf": pod has unbound immediate PersistentVolumeClaims

carlosedp commented 4 years ago

Have you enabled persistence? How's your vars.jsonnet?

charlie-haley commented 4 years ago

Yep, persistence has been enabled, here's a snppet of my config: 

  k3s: {
    enabled: true,
    master_ip: ['192.168.1.111'],
  },

  enablePersistence: {
    prometheus: true,
    grafana: true,
    prometheusPV: '',
    grafanaPV: '',
    storageClass: '',
    prometheusSizePV: '2Gi',
    grafanaSizePV: '20Gi',
  },
carlosedp commented 4 years ago

You need to check if you have a StorageClass provider for dynamic PVs. Also check if your PVs are bound to PVCs. This is usually a problem in the infrastructure.

cgiraldo commented 4 years ago

I have the same problem.

I think we need to set storageClass: null to use the default storageClassName (If storageClassName is not specified in the PVC, the default storage class will be used for provisioning).

If storageClassName is set to an empty string (‘’) in the PVC, no storage class will be used (i.e.; dynamic provisioning is disabled for this PVC).

However setting storageClass: null produces a compilation error:

RUNTIME ERROR: Unexpected type null, expected object
     base_operator_stack.jsonnet:(124:7)-(130:133)  object <anonymous>
carlosedp commented 4 years ago

Try to put null in single quotes.

cgiraldo commented 4 years ago

Tryin with null in single quotes does not work:

my vars.jsonnet:

....
  enablePersistence: {
    // Setting these to false, defaults to emptyDirs.
    prometheus: true,
    grafana: true,
    // If using a pre-created PV, fill in the names below. If blank, they will use the default StorageClass
    prometheusPV: '',
    grafanaPV: '',
    // If required to use a specific storageClass, keep the PV names above blank and fill the storageClass name below.
    storageClass: 'null',
    // Define the PV sizes below
    prometheusSizePV: '2Gi',
    grafanaSizePV: '20Gi',
  },
...

example of compiled manifest (grafana-storage.yaml):

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-storage
  namespace: monitoring
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: "null"
  volumeName: ""

kubectl -n monitoring get pvc
NAME                                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
grafana-storage                      Pending                                      null           2m31s
prometheus-k8s-db-prometheus-k8s-0   Pending                                      null           2m25s
YushchenkoAndrew commented 4 years ago

I have similar issue, don't know how exactly should I declare pv 

 $ kubectl get pods -n monitoring
 NAME                                   READY   STATUS    RESTARTS   AGE
 alertmanager-main-0                    2/2     Running   0          99s
 arm-exporter-8dmsf                     2/2     Running   0          99s
 arm-exporter-drrmm                     2/2     Running   0          99s
 arm-exporter-zmlhh                     2/2     Running   0          99s
 grafana-6ccccd4858-lx854               0/1     Pending   0          96s
 kube-state-metrics-96bf99844-hnnhc     3/3     Running   0          96s
 node-exporter-lbq2b                    2/2     Running   0          95s
 node-exporter-trp9z                    2/2     Running   0          96s
 node-exporter-zbqwh                    2/2     Running   0          95s
 prometheus-adapter-f78c4f4ff-gmw67     1/1     Running   0          95s
 prometheus-k8s-0                       0/3     Pending   0          92s
 prometheus-operator-6b8868d698-fxp5w   2/2     Running   0          116s

pv file for grafana

 apiVersion: v1
 kind: PersistentVolume
 metadata:
   name: grafana
 spec:
   capacity:
     storage: 10Gi
   accessModes:
     - ReadWriteMany
   persistentVolumeReclaimPolicy: Retain
   nfs:
     server: 192.168.0.105
     path: /media/sda1/grafana

 $ kubectl -n monitoring get pvc
 NAME                                 STATUS    VOLUME       CAPACITY   ACCESS MODES   STORAGECLASS   AGE
 grafana-storage                      Pending   grafana      0                         nfs-ssd1       4m37s
 prometheus-k8s-db-prometheus-k8s-0   Pending   prometheus   0                         nfs-ssd1       4m33s

 $ kubectl get pv
 NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
 grafana      10Gi       RWX            Retain           Available                                   34m
 prometheus   2Gi        RWX            Retain           Available                                   34m

 $ kubectl describe pvc grafana-storage -n monitoring
 Name:          grafana-storage
 Namespace:     monitoring
 StorageClass:  nfs-ssd1
 Status:        Pending
 Volume:        grafana
 Labels:        <none>
 Annotations:   Finalizers:  [kubernetes.io/pvc-protection]
 Capacity:      0
 Access Modes:
 VolumeMode:    Filesystem
 Mounted By:    grafana-6ccccd4858-lx854
 Events:
   Type     Reason          Age                  From                         Message
   ----     ------          ----                 ----                         -------
   Warning  VolumeMismatch  0s (x25 over 5m57s)  persistentvolume-controller  Cannot bind to requested volume "grafana": storageClassName does not match

 $ kubectl describe pod prometheus-k8s-0 -n monitoring
 Name:           prometheus-k8s-0
 Namespace:      monitoring
 Priority:       0
 Node:           <none>
 Labels:         app=prometheus
            controller-revision-hash=prometheus-k8s-6bf6ddbb96
            prometheus=k8s
            statefulset.kubernetes.io/pod-name=prometheus-k8s-0

 ............
   Type     Reason            Age        From               Message
   ----     ------            ----       ----               -------
   Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "prometheus-k8s-0": pod has unbound immediate PersistentVolumeClaims
   Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "prometheus-k8s-0": pod has unbound immediate PersistentVolumeClaims

And vars.jsonnet:

// Persistent volume configuration
enablePersistence: {
  // Setting these to false, defaults to emptyDirs.
  prometheus: true,
  grafana: true,
  // If using a pre-created PV, fill in the names below. If blank, they will use the default StorageClass
  prometheusPV: 'prometheus',
  grafanaPV: 'grafana',
  // If required to use a specific storageClass, keep the PV names above blank and fill the storageClass name below.
  storageClass: 'nfs-ssd1',
  // Define the PV sizes below
  prometheusSizePV: '2Gi',
  grafanaSizePV: '10Gi',
},
carlosedp commented 4 years ago

If you just want Kubernetes to create the PV for you using the default (or any specific storageClass), don't specify the PV names (prometheusPV and grafanaPV variables). Leave them blank.

YushchenkoAndrew commented 4 years ago

I tried to live PV and storage Class empty but unfortunately I have such error:

 $ kubectl get pvc -n monitoring
 NAME                                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
 grafana-storage                      Pending                                                     6m8s
 prometheus-k8s-db-prometheus-k8s-0   Pending                                                     6m5s

 $ kubectl describe pvc grafana-storage -n monitoring
 Name:          grafana-storage
 Namespace:     monitoring
 StorageClass:
 Status:        Pending
 Volume:
 Labels:        <none>
 Annotations:   Finalizers:  [kubernetes.io/pvc-protection]
 Capacity:
 Access Modes:
 VolumeMode:    Filesystem
 Mounted By:    grafana-6ccccd4858-cwvqg
 Events:
   Type    Reason         Age                   From                         Message
   ----    ------         ----                  ----                         -------
   Normal  FailedBinding  31s (x26 over 6m31s)  persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

 $ kubectl describe pod grafana-6ccccd4858-cwvqg -n monitoring
 Name:           grafana-6ccccd4858-cwvqg
 Namespace:      monitoring
 Priority:       0
 Node:           <none>
 Labels:         app=grafana
                 pod-template-hash=6ccccd4858
 .......
 Events:
   Type     Reason            Age        From               Message
   ----     ------            ----       ----               -------
   Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "grafana-6ccccd4858-cwvqg": pod has unbound immediate PersistentVolumeClaims
   Warning  FailedScheduling  <unknown>  default-scheduler  running "VolumeBinding" filter plugin for pod "grafana-6ccccd4858-cwvqg": pod has unbound immediate PersistentVolumeClaims

I also tried to apply storageClass that I found in one of your repositories (https://github.com/carlosedp/kubernetes-arm/tree/master/3-NFS_Storage) and one of your tutorial (https://itnext.io/building-an-arm-kubernetes-cluster-ef31032636f9) but I got another errors:

 $ kubectl get pvc -n monitoring
 NAME                                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
 grafana-storage                      Pending                                      nfs-ssd1       31s
 prometheus-k8s-db-prometheus-k8s-0   Pending                                      nfs-ssd1       27s

 $ kubectl describe pvc grafana-storage -n monitoring
 Name:          grafana-storage
 Namespace:     monitoring
 StorageClass:  nfs-ssd1
 Status:        Pending
 Volume:
 Labels:        <none>
 Annotations:   volume.beta.kubernetes.io/storage-provisioner: nfs-provisioner/nfs-ssd1
 Finalizers:    [kubernetes.io/pvc-protection]
 Capacity:
 Access Modes:
 VolumeMode:    Filesystem
 Mounted By:    grafana-6ccccd4858-rwh8l
 Events:
   Type    Reason                Age               From                         Message
   ----    ------                ----              ----                         -------
   Normal  ExternalProvisioning  6s (x5 over 61s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "nfs-provisioner/nfs-ssd1" or manually created by system administrator

Do you have any though what wrong ?

carlosedp commented 4 years ago

You need to have the StorageClass provisioner backend. In this case it's my test cluster NFS. Try without persistence and start improving and debugging what is required..

carlosedp commented 4 years ago

Persistence error. Apparently in the backend.