search

bottlerocket-os / bottlerocket-ecs-updater

A service to automatically manage Bottlerocket updates in an Amazon ECS cluster.
Other
47 stars 9 forks source link

Add mindful draining #43

Open WilboMo opened 3 years ago

WilboMo commented 3 years ago

What I'd like: Currently Updater operates under the assumption that it is the only actor manipulating the instances to DRAINING or ACTIVE. In situations where this is not true instance state could be unintentionally changed negatively by Updater. Implementing a mechanism to check for state changes by other actors and handling the instances accordingly could prevent these issues and allow Updater to run in clusters where it is not the only manipulator of an instance's state.

umairishaq commented 3 years ago

We could use CloudWatch Rule to filter and detect ECS events emitted by CloudTrail. From the various integrations available, like SQS, SNS, Lambda etc., the rule, when triggered, would set off custom logic within Updater. Thoughts?

WilboMo commented 3 years ago

We could use CloudWatch Rule to filter and detect ECS events emitted by CloudTrail. From the various integrations available, like SQS, SNS, Lambda etc., the rule, when triggered, would set off custom logic within Updater. Thoughts?

I think thats a great idea and totally worth exploring. Thank you for the suggestion!

ybob1980 commented 4 weeks ago

Could we extend ECS updater to cover non-service task as well? For example,

(1) ECS updater put instance in drain mode to prevent further task placement (2) After waiting for x minutes (configurable value, timeout) or existing running tasks complete, reboot instance to consume the update