search

phillbaker / terraform-provider-mailgunv3

mailgun provider for terraform based on v3 of the api
MIT License
10 stars 3 forks source link

Feature Request: Handle MailGun's Rate limit #15

Closed nicoleyson closed 5 years ago

nicoleyson commented 5 years ago 
Note Mailgun imposes a rate limit for the Domains API endpoint. 
Users may issue no more than 300 requests per minute, per account. 
See the resultant rate limit response below.

source

I've temporarily resolved this issue by setting -parallelism=0 but it's slowing down the whole run (e.g. providers that are consuming the mailgun resources such as AWS's route 53 resources)

Is there a way to implement a back off / slowdown the number of requests the provider makes since we know what the rate limit is ahead of time? I believe the AWS provider does something similar to avoid hitting the AWS rate limit.

phillbaker commented 5 years ago

Hm, interesting - how many domains are you managing? Just curious at what level this becomes an issue.

nicoleyson commented 5 years ago

It became an issue around 50. At that point I set parallelism to 5, which resolved it temporarily. When it increased to 150 domains, I set parallelism to 3.

We are currently at 470 and now parallelism is set to either 1 or 0 😂 It usually fails when it's refreshing state. 

* module.REDACTED_COM.mailgunv3_domain.acquired: mailgunv3_domain.acquired: Error retrieving domain: UnexpectedResponseError URL=https://api.mailgun.net/v3/domains/REDACTED.com ExpectedOneOf=[]int{200, 202, 204} Got=429 Error: max rate reached: retry-in 200ms
phillbaker commented 5 years ago

Interesting - that's quite a few domains 😀 .

In the AWS case it looks like it was solved by the upstream AWS library: https://github.com/hashicorp/terraform/issues/1051.

I've opened an issue with the upstream library to clarify: https://github.com/mailgun/mailgun-go/issues/199

nicoleyson commented 5 years ago

Oh nice find! When I was digging through the AWS provider issues, they seemed to indicate that it was resolved in the provider.

Thanks! Sorry for the misinformation! I know the maintainer for mailgun-go is super responsive, so I have no doubt this will be resolved soon. Thanks for pushing this forward.

phillbaker commented 5 years ago

Looks like it'll be on the provider to handle this for the time being - I've got something that I think works, but writing some tests to confirm.

nicoleyson commented 5 years ago

That is amazing!!!! You are a life saver!!! Thank you!!!!!!!!

phillbaker commented 5 years ago

should be closed with b571775f9492db861b71b8f45d345724ce03bc77, released https://github.com/phillbaker/terraform-provider-mailgunv3/releases/tag/v0.3.2

nicoleyson commented 5 years ago

Awesome!! I'll bump our provider! 🎉 🎉 🎉 🎉 🎉 🎉 🎉 Thank you so so much for handling this. You really saved us.

nicoleyson commented 5 years ago

Confirmed it's working. I took parallelism off 0, so it's defaulting to 10 now, I'd imagine and it's working perfectly. Thank you!!!!