stockparfait / stockparfait

The core engine of Stock Parfait, command line public version.
Apache License 2.0
1 stars 0 forks source link

Implement quota tracking for Nasdaq Data Link #20

Open sergey-a-berezin opened 2 years ago

sergey-a-berezin commented 2 years ago

Nasdaq Data Link API has quota limits of 5000 requests / 10 minutes for regular requests, 720K requests per day, and 10 bulk download requests per hour for premium subscribers.

These limits should be the default, but also parameterizable so they can be used by non-premium or even unauthenticated users, who have different rate limits.

sergey-a-berezin commented 2 years ago

In general, quotas should be specified as a list of [number of calls, per time duration]. At the moment, it is not clear how these quotas are enforced exactly. E.g. when the 720K calls per day are refreshed: at 12am in some time zone (New York?), or 24h after the first call? Similarly for the 5K per 10 min period - is it tied to the absolute time, or the time elapsed since the first relevant call?

To make it simple, we assume that it is the elapsed time, and will adjust later if necessary.

The quota tracker will have the following go-routine-safe API:

This API allows for efficient parallel processing using limited number of parallel workers. For instance, 10 workers can reserve 5000 calls each (for the total of 5K), execute them in parallel, sleep for WaitFor() time, and request quota for the next batch.