grafana / k8s-monitoring-helm

Apache License 2.0
178 stars 72 forks source link

prometheus-node-exporter bypasses all taints #705

Closed AlissonRS closed 1 week ago

AlissonRS commented 1 month ago

I have a few nodes with taints so it only schedules pods I want to.

The node-exporter has the toleration below:

    - operator: Exists
      effect: NoSchedule

Which seems to be a "match all keys" toleration, essentially bypassing my taints.

Is there a way to override this, or rather remove it from the chart as default and keep only the ones with specific keys?

boniek83 commented 2 weeks ago

As a workaround you can deploy helm charts using kustomize which in turn allows patching of resources to your liking.

petewall commented 1 week ago

Have you tried setting:

prometheus-node-exporter:
  tolerations:
  - <SOME OTHER TOLERATION RULE>

Helm doesn't merge arrays if you specify that in your values file.

Note that Node Exporter does this because it works by being on every node. Without this toleration, you'll lose node metrics for that node. I totally acknowledge that there are reasons to do that, but be aware of the tradeoff.