Given that dynamically allocated executors (from k8s-spark-scheduler's point of view at least) are scheduled one-by-one, the behavior is rather simple:
if we can find a node that already has pods belonging to this app, and enough capacity for the pod being scheduled, then use it. If there are multiple nodes fitting this criteria, pick the 'fullest' one.
otherwise just find a prefered node with small capacity (fairly similar to tightly pack)
Explicitly this means that if 10 dynamically allocated executors appear, and there's no available space on nodes hosting this spark app, then those 10 pods will likely be more fragmented than if we could schedule all 10 of them at once.
svc-autorelease
commented
1 year ago
Given that dynamically allocated executors (from k8s-spark-scheduler's point of view at least) are scheduled one-by-one, the behavior is rather simple:
Explicitly this means that if 10 dynamically allocated executors appear, and there's no available space on nodes hosting this spark app, then those 10 pods will likely be more fragmented than if we could schedule all 10 of them at once.