Following on from discussions in dask/distributed#3300 it would be great if workers gracefull closed when they recieve a spot termination notification. This will be even more useful with #47.
ECSCluster and FargateCluster workers should start with a --preload 'dask_cloudprovider.aws.spot_termination_handler' or similar. This could also be used for manual cluster creation separately from close cluster managers.
This new dask_cloudprovider.aws.spot_termination_handler module should attach a task to the workers periodic_callbacks to poll the Spot Termination endpoint.
When it gets a spot termination notice it should call close_gracefully on the worker.
jacobtomlinson
commented
2 years ago
Following on from discussions in dask/distributed#3300 it would be great if workers gracefull closed when they recieve a spot termination notification. This will be even more useful with #47.
ECSClusterandFargateClusterworkers should start with a--preload 'dask_cloudprovider.aws.spot_termination_handler'or similar. This could also be used for manual cluster creation separately from close cluster managers.dask_cloudprovider.aws.spot_termination_handlermodule should attach a task to the workersperiodic_callbacksto poll the Spot Termination endpoint.close_gracefullyon the worker.