search

NixOS / nixops-aws

GNU Lesser General Public License v3.0
52 stars 38 forks source link

Rate limits #94

Open coretemp opened 6 years ago

coretemp commented 6 years ago

NixOps doesn't cope with rate limits.

boto.exception.BotoServerError: BotoServerError: 400 Bad Request
<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-08/">
  <Error>
    <Type>Sender</Type>
    <Code>Throttling</Code>
    <Message>Rate exceeded</Message>
  </Error>

NixOps could output the amount of requests being done to the various APIs and produce a report on such failure which could then be sent to AWS support immediately.

In my case running the exact same command (nixops deploy) again worked.

NixOps could also try a bit harder. E.g. run three times in a row with 5 seconds in between before raising the white flag.

kaii-zen commented 6 years ago

Hitting this sometimes as well. Sometimes it's enough to just run the same command again immediately but other times it's a forced coffee break. I feel like it happens more often in NixOps than in Terraform for example but that's hardly a scientific observation 😅

cleverca22 commented 6 years ago

some services will also report how many hits you have done, how many remain in your quota, and when it resets, in an http header, and you can then limit yourself to avoid hitting them, that could also be looked into

but a quick glance at the aws headers shows no such tips being hidden in the reply

coretemp commented 6 years ago

This issue is blocked by https://github.com/NixOS/nixops/issues/504.