Graceful handling of API limits for large resource sets

[quinncomendant]

Rackspace Cloud DNS, like all providers, imposes limits on the frequency of API requests. E.g., 60/minute for GET /domains. I'm hitting this limit when running denominator zone list:

{q@oak/0 ~} sudo /root/bin/denominator -C /root/etc/denominator.yml -n clouddns-strangecode zone list >| strangecode-zones-20160923.txt
[CloudIdentity#apiKeyAuth] ---> POST https://identity.api.rackspacecloud.com/v2.0/tokens HTTP/1.1
[CloudIdentity#apiKeyAuth] <--- HTTP/1.1 200 OK (614ms)
[CloudDNS#domains] ---> GET https://dns.api.rackspacecloud.com/v1.0/721416/domains HTTP/1.1
[CloudDNS#domains] <--- HTTP/1.1 200 OK (924ms)
[CloudDNS#recordsByNameAndType] ---> GET https://dns.api.rackspacecloud.com/v1.0/721416/domains/3794571/records?name=1078gallery.org&type=SOA HTTP/1.1
[CloudDNS#recordsByNameAndType] <--- HTTP/1.1 200 OK (125ms)

[…lots of output…]

[CloudDNS#recordsByNameAndType] ---> GET https://dns.api.rackspacecloud.com/v1.0/721416/domains/3795191/records?name=aandeautomotive.net&type=SOA HTTP/1.1
[CloudDNS#recordsByNameAndType] <--- HTTP/1.1 413 Request Entity Too Large (42ms)
[CloudDNS#recordsByNameAndType] ---> RETRYING
[CloudDNS#recordsByNameAndType] ---> GET https://dns.api.rackspacecloud.com/v1.0/721416/domains/3795191/records?name=aandeautomotive.net&type=SOA HTTP/1.1
[CloudDNS#recordsByNameAndType] <--- HTTP/1.1 413 Request Entity Too Large (42ms)
[CloudDNS#recordsByNameAndType] ---> RETRYING
[CloudDNS#recordsByNameAndType] ---> GET https://dns.api.rackspacecloud.com/v1.0/721416/domains/3795191/records?name=aandeautomotive.net&type=SOA HTTP/1.1
[CloudDNS#recordsByNameAndType] <--- HTTP/1.1 413 Request Entity Too Large (42ms)
;; error: status 413 reading CloudDNS#recordsByNameAndType(int,String,String); content:
{
    "overLimit" : {
        "code" : 413,
        "message" : "OverLimit Retry...",
        "details" : "Error Details...",
   "retryAfter" : "2016-09-23T21:48:37Z"
    }
}

Because denominator cannot handle this API limit, there is no way for me to list all zones. 😨

I would expect denominator to pause occasionally to avoid the OverLimit Retry error.

[quinncomendant]

In the meantime, I've hacked up an ungraceful line of bash to pause denominator every so many requests to prevent overrunning the API limits. Maybe useful for others with the same problem (caveat: it will pause all processes matching pidof java). It works by watching the output file, and every 3 zones that are saved, it will pause for 4 seconds (so at most 45 zones/minute are retrieved). UPDATE: a 25 second sleep was too slow and the API would timeout; lowered to 4 second sleep every 3 domains.

denominator zone list >| allzones.txt & sleep 2; DPID=$(pidof java); echo "I will pause PID '$DPID' for 4 sec every 3 zones"; while sudo kill -0 $DPID 2>/dev/null; do if [[ $(($(wc -l < allzones.txt) % 3)) -eq 0 ]]; then sudo kill -STOP $DPID; sleep 2; sudo kill -CONT $DPID; fi; done

Obviously, it'd still be great to have this functionality built-in to denominator.

[quinncomendant]

I packaged the previous hack into a script (requires customizing before running). https://gist.github.com/quinncomendant/c65cf0c768812c8cc429876603f5c5c7