In the DeamonSet yaml the FluentBit pod has blanket tolerations, so EKS tried to schedule them on Fargate nodes

[ottokruse]

Actual

I've installed CloudWatch Container Insights per the DaemonSet QuickStart.

This creates 2 DeamonSets, 1 for CloudWatch agent and 1 for FluentBit. I notice that Kubernetes tries to schedule the Fluentbit DaemonSet pod also on (pre-existing) Fargate nodes, which obviously does not work (the pods stay pending forever). Interestingly, no pods are created against Fargate nodes for the CloudWatch daemonset (which is good).

Expected

FluentBid pods are not scheduled against Fargate nodes (as that is not supported on Fargate anyway, these pods stay pending forever).

If I want to run FluentBit in Fargate, I'll use a sidecar container for it.

Root cause

At the bottom of DaemonSet QuickStart there's the pod spec for FluentBit that has these tolerations:

      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

The last 2 tolerations have no key and are thus blanket tolerations, triggering the scheduling on (pre-existing) Fargate nodes.

Note that the pod spec for the CloudWatch agent in the same yaml doesn't have any tolerations.