Support for Pod Topology Spread Constraints

[xuzeng012]

Pod Topology Spread Constraints are important for enhancing high availability and fault tolerance by ensuring that pods are distributed across different topologies, such as nodes in different availability zones or failure domains. This feature would be highly beneficial for users who want to deploy Spark workloads in multi-zone or multi-cluster environments to improve reliability and resilience. Users should be able to configure constraints such as maxSkew, minDomains, topologyKey, and whenUnsatisfiable for Spark pods to ensure they are spread across different nodes and failure domains.

[ruhz3]

Spark Operator do support podAntiAffinity, so this is probably the best we can do for now.

spec:
  executor:
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: spark-app-name
                operator: In
                values:
                - {{ your service name }}
            topologyKey: kubernetes.io/hostname

[korjek]

@ruhz3 podAntiAffinity can't be used as a replacement for TSC. For example, if you want to run 10 executor pods on at least two nodes, this can't be achieved with affinity.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.