grafana / helm-charts

Apache License 2.0
1.65k stars 2.27k forks source link

How to specific affinity for loki? #2709

Open jenia opened 1 year ago

jenia commented 1 year ago

Hello.

I'm installing Loki to Kubernetes. I'm using minikube on my PC right now :blush: and Loki fails to starts because 2/3 of the loki-write pods have affinity errors:

loki-backend-0                                0/2     Pending   0          85m
loki-backend-1                                2/2     Running   0          85m
loki-backend-2                                0/2     Pending   0          85m
loki-canary-8lmxf                             1/1     Running   0          85m
loki-gateway-5f96d67f97-2mxls                 1/1     Running   0          85m
loki-grafana-agent-operator-d7c684bf9-jqqd8   1/1     Running   0          85m
loki-logs-vnlkk                               2/2     Running   0          85m
loki-read-746d977fdd-d94wz                    1/1     Running   0          85m
loki-read-746d977fdd-smfxb                    0/1     Pending   0          85m
loki-read-746d977fdd-z57z7                    0/1     Pending   0          85m
loki-write-0                                  0/1     Pending   0          85m
loki-write-1                                  1/1     Running   0          85m
loki-write-2                                  0/1     Pending   0          85m

kubectl describe pod loki-write-1
1 node(s) didn't match pod anti-affinity rules. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.. 

How do I specify to the this affinity rule for all loki pods:

nodeSelector:
    kubernetes.io/hostname: minikube

P.S. I know I can use monolith Loki but the problem is that it doesn't support minio but only the filesystem as storage. So if possible, I'd like to use the scalable Loki with minio but run it all on one node.

Davenury commented 10 months ago

I guess this is the chart you're using. If so, you can override default affinity for write component here, by simply passing an empty object:

write:
  ...
  affinity: {}

and it should work (you'll probably need to override it everywhere it's used),

dedeira commented 2 weeks ago

Did work? for me, using grafana/loki 6.18.0 it did not. I tried with affinity: {}

affinity: null

 affinity:
    podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: write
              topologyKey: kubernetes.io/hostname

For first two cases I got the default value as if nothing happened. For the last I got both values merged:

spec:
     affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
             - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: write
              topologyKey: kubernetes.io/hostname
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app.kubernetes.io/component: write
              topologyKey: kubernetes.io/hostname

It seems it does a merge of the values, not an override as expected. So for now I just declared the same affinity but with a bogus component "nonexistent": affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution:

This seems silly though. If someone finds a more elegant solution, it would be nice.