Open zaldnoay opened 1 year ago
We had exactly the same issue.
We wanted Loki to scale/downscale more steadily by tuning both the behavior.scaleUp
and behavior.scaleDown
policies, but we couldn't using the provided HPA resources, so we rolled out our own manifests on top of the chart.
One of the problems we had is that unless we enable HPA with autoscaling.enabled: true
, which we don't want to given that we use our own HPA manifests, we can't avoid setting the replicas
of each component.
spec:
{{- if not .Values.distributor.autoscaling.enabled }}
replicas: {{ .Values.distributor.replicas }}
{{- end }}
That's a problem when using a GitOps operator like Argo CD, because once the HPA tries to scale, Argo CD will reconcile the state setting whatever the value is in the replicas
option, preventing any scale up.
We solved it by ignoring that field in Argo CD but it'll be nice to be able to use custom HPAs configurations or KEDA objects, and still be able to avoid defining the replica
in the templates.
ignoreDifferences:
- group: apps
kind: Deployment
name: loki-distributor
namespace: loki
jsonPointers:
- /spec/replicas
- group: apps
kind: StatefulSet
name: loki-ingester
namespace: loki
jsonPointers:
- /spec/replicas
- group: apps
kind: Deployment
name: loki-querier
namespace: loki
jsonPointers:
- /spec/replicas
- group: apps
kind: Deployment
name: loki-query-frontend
namespace: loki
jsonPointers:
- /spec/replicas
syncPolicy:
syncOptions:
- RespectIgnoreDifferences=true
Loki's document recommend using KEDA in querier to configure autoscaling based on Prometheus metrics. Also the default scaling behaviour is too frequent for Loki's components. I recommend adding a configurable scaling behaviour to the values and templates to make deployment more stable and flexible. Here are some of the examples I wrote:
values.yaml:
templates:
Questions are welcome.
KEDA document: https://keda.sh/docs/latest/concepts/scaling-deployments/ K8S HPA document: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#default-behavior