alantech / iasql

Cloud Infrastructure as data in PostgreSQL
https://iasql.com
MIT License
591 stars 24 forks source link

Eventually consistent AWS APIs #1059

Open aguillenv opened 2 years ago

aguillenv commented 2 years ago

Some AWS APIs seem to be eventually consistent.

In this specific case https://github.com/iasql/iasql-engine/runs/7635873948?check_suite_focus=true#step:5:13889, the test fail because:

  1. we call the API to update tags
  2. then we called the ebs API to get volume updated (it should include tags) but tags did not come updated
  3. we saved in db the entity with outdated tags
  4. go to a new loop to update tags, removing again from cloud the newly updated
  5. the test failed
dfellis commented 2 years ago

A potential solution:

  1. For tag updates or any other eventually-consistent API, we first make the change as normal, and if it is successful then we sleep for a brief amount of time (say 250ms)
  2. Enter a do-while query loop where we query the tags again and check if they are equal. If not, we sleep a longer amount of time (potentially in an exponential back-off approach to minimize average latency without a ton of read calls) and repeat, until some maximum timeout where we fail out the change.