search

RhodiumGroup / rhg_compute_tools

Tools for using compute.rhg.com and compute.impactlab.org
MIT License
1 stars 4 forks source link

spin down workers as tasks finish #24

Closed delgadom closed 5 years ago

delgadom commented 5 years ago

Description

for large client.map workflows with long-running tasks, would be nice to have a process we could spin off to shut down workers as tasks complete when there are no tasks waiting and there are workers sitting idle

delgadom commented 5 years ago

something along the lines of

from time import sleep
check_interval = 60 #seconds

def scale_down_workers_eventually(cluster, ftrs, check_interval):
    n_workers = len(cluster.pods())
    n_pending = len([f for f in ftrs if f.status == 'pending'])
    while n_pending > 0:
        n_pending = len([f for f in ftrs if f.status == 'pending'])
        if n_pending < n_workers:
            cluster.scale(n_pending)
            sleep(check_interval)
    return None