Calico Kube Controller Deployment Not Scheduled on System Pools

JasonWhall commented 3 years ago

What happened: Since the update to Calico 3.18.x there have been new deployments added to the cluster for Calico use. Most of these use an affinity rule to schedule the pods on a "System" node pool. For the calico-kube-controllers deployment this affinity rule is not applied

What you expected to happen: I would expect the calico-kube-controllers deployment to use the affinity rule similar to other deployments to prefer scheduling on a system node pool. Example below from the calico-typha deployment:

      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - preference:
              matchExpressions:
              - key: kubernetes.azure.com/mode
                operator: In
                values:
                - system
            weight: 100
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-node
              - key: kubernetes.azure.com/cluster
                operator: Exists

How to reproduce it (as minimally and precisely as possible): Deploy AKS cluster 1.20.5 and check calico-kube-controllers deployment in calico-system namespace

Anything else we need to know?: AFAICT This can potentially stop "User" node pools auto-scaling down to 0 nodes as it has no preference to be re-scheduled elsewhere.

Environment:

Kubernetes version (use kubectl version): 1.20.5
Size of cluster (how many worker nodes are in the cluster?)
General description of workloads in the cluster (e.g. HTTP microservices, Java app, Ruby on Rails, machine learning, etc.)
Others: