quintilesims / layer0

Build, Manage, and Deploy Your Applications
Apache License 2.0
44 stars 20 forks source link

Add retry logic for eventually consistent resources #569

Closed diemonster closed 6 years ago

diemonster commented 6 years ago

When creating and deleting resources in Layer0, the AWS API often quickly returns a response designating that they have received the request, but that request is not necessarily reflective of reality. AWS terms this as 'eventual consistency', and we must build retry logic in order to account for it.

ELBs and Security Groups are the biggest offenders here --- but other AWS resources have the potential to be eventually consistent as well. As such, we should build middleware in the API's client package that checks if a resource has been created or deleted before returning.