Open sergey-a-berezin opened 2 years ago
In general, quotas should be specified as a list of [number of calls, per time duration]. At the moment, it is not clear how these quotas are enforced exactly. E.g. when the 720K calls per day are refreshed: at 12am in some time zone (New York?), or 24h after the first call? Similarly for the 5K per 10 min period - is it tied to the absolute time, or the time elapsed since the first relevant call?
To make it simple, we assume that it is the elapsed time, and will adjust later if necessary.
The quota tracker will have the following go-routine-safe API:
Reserve(n)
- reserve at most n
calls, or however many are available. It will return an opaque handle which identifies the moment in time, for correct releasing of these calls later.Release(n, handle)
- release previously reserved n
calls. This makes these calls immediately available within the same quota limits, if they didn't yet expire. Releasing calls only adjusts the current quotas up to their maximum limits.WaitFor()
- returns the time.Duration
to wait before more quota is available. If the quota is still available now, the value is 0.This API allows for efficient parallel processing using limited number of parallel workers. For instance, 10 workers can reserve 5000 calls each (for the total of 5K), execute them in parallel, sleep for WaitFor()
time, and request quota for the next batch.
Nasdaq Data Link API has quota limits of 5000 requests / 10 minutes for regular requests, 720K requests per day, and 10 bulk download requests per hour for premium subscribers.
These limits should be the default, but also parameterizable so they can be used by non-premium or even unauthenticated users, who have different rate limits.