search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
23.73k stars 3.43k forks source link

Storage resizing doesn't work with SingleBinary deployment. #12854

Open mindthecap opened 5 months ago

mindthecap commented 5 months ago

Storage resize in SingleBinary deployment is impossible because it's created with StatefulSets, which prohibits changing the manifest after deployment. The affected line in code: https://github.com/grafana/loki/blob/599a3002d2878eafd2a64dc270f662f9756cae5b/production/helm/loki/templates/single-binary/statefulset.yaml#L172-L191

SingleBinary deployment should create separate PVC and reference it in the StatefulSet.

mindthecap commented 5 months ago

A workaround for this:

  1. Resize the PVC and PV.
  2. Delete the statefulset but keep the pod kubectl delete sts --cascade=orphan loki.
  3. Redeploy Loki with helm.
  4. Delete the pod to trigger volume resizing.

More info in here: https://serverfault.com/questions/955293/how-to-increase-disk-size-in-a-stateful-set