search

syncloud / platform

Run popular services on your device with one click
https://syncloud.org
GNU General Public License v3.0
401 stars 40 forks source link

redirect: Amazon Route53 request rate limit (5 per second) #255

Closed cyberb closed 7 years ago

cyberb commented 7 years ago

Need to think about reducing (optimizing) calls per second to AWS Route53 (dns) as they have hard limit of 5 per second: http://docs.aws.amazon.com/Route53/latest/DeveloperGuide/DNSLimitations.html

One of the solutions is:

  1. Not update Route53 from every client update instead just update DB.
  2. Have separate sync process to update Route53 with stable rate of requests.
  3. Think about the same potential problem in future with DB (now feels more than fine)
vsapronov commented 7 years ago

I think we update Route53 only on changes of ip addresses. https://github.com/syncloud/redirect/blob/master/redirect/services.py#L207

On Sun, May 28, 2017 at 5:46 PM, Boris Rybalkin notifications@github.com wrote:

Need to think about reducing (optimizing) calls per second to AWS Route53 (dns) as they have hard limit of 5 per second: http://docs.aws.amazon.com/ Route53/latest/DeveloperGuide/DNSLimitations.html

One of the solutions is:

  1. Not update Route53 from every client update instead just update DB.
  2. Have separate sync process to update Route53 with stable rate of requests.
  3. Think about the same potential problem in future with DB (now feels more than fine)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/syncloud/platform/issues/255, or mute the thread https://github.com/notifications/unsubscribe-auth/AA2FtpfH9eSBEMweM5LKzGqlECu2qtoJks5r-es-gaJpZM4NoxTE .

--

Vladimir Sapronov

cyberb commented 7 years ago

You are right, I forgot about it. After checking one more time our error stack traces, I have found that it is coming from domain delete method:

https://github.com/syncloud/redirect/blob/master/redirect/dns.py#L44

Probably boto library has some iterator there.


[Sun May 28 11:19:35.467374 2017] [:error] [pid 9662] <?xml version="1.0"?>
[Sun May 28 11:19:35.467382 2017] [:error] [pid 9662] <ErrorResponse xmlns="https://route53.amazonaws.com/doc/2013-04-01/"><Error><Type>Sender</Type><Code>Throttling</Code><Message>Rate exceeded</Message></Error><RequestId>87b57bea-4397-11e7-86d9-ff50ff269951</RequestId></ErrorResponse>
[Sun May 28 11:19:35.493653 2017] [:error] [pid 9662] ERROR:root:exception happened
[Sun May 28 11:19:35.493675 2017] [:error] [pid 9662] Traceback (most recent call last):
[Sun May 28 11:19:35.493684 2017] [:error] [pid 9662]   File "/var/www/redirect-prod/redirect/services.py", line 228, in user_domain_delete
[Sun May 28 11:19:35.493692 2017] [:error] [pid 9662]     self.dns.delete_domain(self.main_domain, domain)
[Sun May 28 11:19:35.493700 2017] [:error] [pid 9662]   File "/var/www/redirect-prod/redirect/dns.py", line 44, in delete_domain
[Sun May 28 11:19:35.493707 2017] [:error] [pid 9662]     if zone.find_records(full_domain, 'A'):
[Sun May 28 11:19:35.493715 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/zone.py", line 236, in find_records
[Sun May 28 11:19:35.493723 2017] [:error] [pid 9662]     results = [r for r in returned if r.name == name and r.type == type]
[Sun May 28 11:19:35.493731 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493739 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493746 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493754 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493761 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493769 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493777 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493784 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493792 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493799 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493807 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493815 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493822 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493830 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493837 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493845 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493853 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493860 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493868 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493875 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493883 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493890 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493898 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493919 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493928 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493935 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493943 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493950 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493957 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493965 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493972 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493979 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.493987 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 182, in __iter__
[Sun May 28 11:19:35.493994 2017] [:error] [pid 9662]     for obj in results:
[Sun May 28 11:19:35.494001 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/record.py", line 186, in __iter__
[Sun May 28 11:19:35.494009 2017] [:error] [pid 9662]     results = self.connection.get_all_rrsets(self.hosted_zone_id, name=self.next_record_name, type=self.next_record_type)
[Sun May 28 11:19:35.494017 2017] [:error] [pid 9662]   File "/usr/local/lib/python2.7/dist-packages/boto/route53/connection.py", line 394, in get_all_rrsets
[Sun May 28 11:19:35.494024 2017] [:error] [pid 9662]     body)
cyberb commented 7 years ago

Removed find_records call on our side before calling delete as boto does the same thing inside delete. https://github.com/syncloud/redirect/commit/867724f7a5a9668df8fd53b03b923b456a6fda40#diff-4b67f31dd5e7c3306d42b771fe84eb35L44

cyberb commented 7 years ago

removed completely usage of find based (multi request) delete and switched to simple (single request) delete action. https://github.com/syncloud/redirect/commit/720308e1e5f621c0d3a378e52f531e24f15d02c5