search

docker-archive / deploykit

A toolkit for creating and managing declarative, self-healing infrastructure.
Apache License 2.0
2.25k stars 262 forks source link

Mitigate the risk of creating a LOT of instances or deleting instances that we don't manage? #409

Open dgageot opened 7 years ago

dgageot commented 7 years ago

If there is a bug in an Infrakit plugin or a misconfiguration in the templates, Infrakit could create too many instances or it could delete instances it doesn't manage or from another group.

How could be mitigate this kind of behaviour with some kind of cross cutting checks/quotas/protection/...?

thebsdbox commented 7 years ago

It's certainly a concern, i've had occasions where the instances I'm provisioning haven't reported complete in time. This results in the group plugin attempting to deploy more instances, then having to delete the excess when they've all finished deploying.

To mitigate this, i've added in logic into the instance plugin so that it will keep track of what is "being" provisioned and allows the plugin to have more control over the actual state. Allowing the plugin to report back to the group plugin that whilst the instances aren't ready, they're being provisioned.