Closed delgadom closed 5 years ago
something along the lines of
from time import sleep
check_interval = 60 #seconds
def scale_down_workers_eventually(cluster, ftrs, check_interval):
n_workers = len(cluster.pods())
n_pending = len([f for f in ftrs if f.status == 'pending'])
while n_pending > 0:
n_pending = len([f for f in ftrs if f.status == 'pending'])
if n_pending < n_workers:
cluster.scale(n_pending)
sleep(check_interval)
return None
Description
for large client.map workflows with long-running tasks, would be nice to have a process we could spin off to shut down workers as tasks complete when there are no tasks waiting and there are workers sitting idle