search

grafana / helm-charts

Apache License 2.0
1.63k stars 2.26k forks source link

Unable to install loki-stack while prometheus.enabled=true #741

Open hkpanchani opened 3 years ago

hkpanchani commented 3 years ago

Config file:

loki:
  enabled: true
  persistence:
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 5Gi
  limits_config:
    retention_period: 744h

promtail:
  enabled: true

grafana:
  enabled: true
  persistence:
    type: statefulset
    enabled: true
    accessModes:
      - ReadWriteOnce
    size: 1Gi
  sidecar:
    datasources:
      enabled: true

prometheus:
  enabled: true

The following error occurs when prometheus.enabled=true is placed inside Values. Error: UPGRADE FAILED: [unable to recognize "": no matches for kind "ClusterRole" in version "rbac.authorization.k8s.io/v1beta1", unable to recognize "": no matches for kind "ClusterRoleBinding" in version "rbac.authorization.k8s.io/v1beta1"]

hkiang01 commented 3 years ago

Ran into this also. If you pull the Helm chart and untar it using helm pull grafana/loki-stack --version 2.5.0 --untar, you'll notice that the source of the issue is that RBAC by default uses an old apiVersion in loki-stack/charts/prometheus/charts/kube-state-metrics/templates/clusterrole.yaml:

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole

Based on kube-state-metrics's Compatibility Matrix, it's using kube-state-metrics version 1.9.7 (as seen in loki-stack/charts/prometheus/charts/kube-state-metrics/Chart.yaml), which is incompatible with many Kubernetes versions:

  • The Kubernetes cluster has features the client-go library can't use (additional API objects, deprecated APIs, etc).

To mitigate this with a quick fix, you can set the following in your values.yaml:

prometheus:
  kubeStateMetrics:
    enabled: false

The downside is you lose installing kube-state-metrics, but at least you'll have your loki-stack.

I would imagine you'd also want to install kube-state-metrics separately from https://artifacthub.io/packages/helm/prometheus-community/kube-state-metrics/ if you want all the nice built in charts

hkiang01 commented 3 years ago

An alternative: kube-prometheus-stack has the update apiVersions

Edit: you can then simply add Loki as a data source in Grafana per https://github.com/prometheus-operator/kube-prometheus/issues/555#issuecomment-635938188