search

mathpaquette / IQFeed.CSharpApiClient

IQFeed.CSharpApiClient is fastest and the most well-designed C# DTN IQFeed socket API connector available
MIT License
120 stars 43 forks source link

IQFeed history request rate limiting change. #92

Closed rsgabriel closed 3 years ago

rsgabriel commented 3 years ago

Are there plans to support the following recent DTN change?

Last weekend, IQFeed released a server-side update to our historical data servers which changed the method that we use to rate limit requests. This change was necessary to protect our servers from rogue software and to help standardize and improve the performance of our history servers for all customers. This change ONLY applies to our history servers.

Previously, customers were allowed to have 15 simultaneous requests active at any given time. While this was a good rate limiting method when it was originally developed, advances in machine hardware and reduced latency to our servers over the internet had effectively removed this limit entirely. Customers were able to make nearly unlimited requests and that was never the intent for our service. This method of request rate limiting was confusing for new customers. It was dependent upon latency to our servers so each customer had a different limit that they could achieve. This was also never intended, and led to mixed results and customer experiences. For years we were able to operate this way using an informal "honor system", by closely monitoring our server performance and reaching out to customers to adjust their software as needed. However, this approach is no longer feasible.

The new method is a straightforward "requests per second" rate limiting method. It is a bucket limit system similar to the Linux kernel's iptables hashlimit. As a result, you start with a full bucket of request credits. Each request consumes 1 credit and credits are re-allocated every few milliseconds unless you are already at max. If you send a request when you are out of requests, you will get an error stating too many requests (the error message is the same as before). This new method allows all customers to have the same limit. Also, customers can monitor the limit themselves, and know their real limit without trial and error

With all that said, the new limit for requests is 50/second (with one new credit added each 20ms). For the vast majority of customers, this change will not affect their IQFeed performance at all. Unfortunately, for a few of you who are downloading a large symbol list as fast as possible, you will notice this change (if you haven't already). Almost all of you who fall into this category can still get the data you need, but it will take you a bit more time to make the requests. If you fall into this category, we urge you to reach out to our dev support team with a description of the data you need and the requests you are currently trying to use to download it. Our support staff will help you figure out the best way to get this data from the feed.

mathpaquette commented 3 years ago

This has been already implemented in version 2.6.0

mathpaquette commented 3 years ago

@rsgabriel code here https://github.com/mathpaquette/IQFeed.CSharpApiClient/blob/master/src/IQFeed.CSharpApiClient/Lookup/LookupRateLimiter.cs#L12

please star the project if you feel thats helpful.