search

mathpaquette / IQFeed.CSharpApiClient

IQFeed.CSharpApiClient is fastest and the most well-designed C# DTN IQFeed socket API connector available
MIT License
120 stars 43 forks source link

No issue - just a Heads-Up - New IQFeed limit on history requests #90

Closed BunkerCoder closed 3 years ago

BunkerCoder commented 3 years ago

Today, I received "E,Too many simultaneous history requests" in response to an HTD request. I called support and they said it is a recent change, and they will be posting it on the developer forum soon. The change was explained to me as: Basically each user has a limit of 30 history requests per second, and every 1/30 of a second an existing request "drops off" and it decreases by one. As long you stay below an average of 30 request/second (and there is a little padding) you'll be fine.

Just wanted to pass it along.

mathpaquette commented 3 years ago

Hi BunkerCoder,

I noticed that also this weekend while doing download for the previous week. Raised support dev issue but didn’t receive update Yet. Thanks for sharing with us... looks like we will need something to prevent that...

Thanks

Sent from my iPhone

On Dec 21, 2020, at 5:41 PM, BunkerCoder notifications@github.com wrote:



Today, I received "E,Too many simultaneous history requests" in response to an HTD request. I called support and they said it is a recent change, and they will be posting it on the developer forum soon. The change was explained to me as: Basically each user has a limit of 30 history requests per second, and every 1/30 of a second an existing request "drops off" and it decreases by one. As long you stay below an average of 30 request/second (and there is a little padding) you'll be fine.

Just wanted to pass it along.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/mathpaquette/IQFeed.CSharpApiClient/issues/90, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABBICNY3GU4QFEQG52MHRNTSV7FILANCNFSM4VE2JTMQ.

mathpaquette commented 3 years ago

@BunkerCoder do you think this will be also required for ticks? Ill be adding required changes to avoid that.

BunkerCoder commented 3 years ago

@mathpaquette Thanks for taking this on. Got this from support: By the way this is only history requests. Port 5009 (Live) will not be changed.

BunkerCoder commented 3 years ago

Also per support: It's on a "user + IP address" basis So the limit is across all clients.

mathpaquette commented 3 years ago

@BunkerCoder created a fix. Not tested yet. Do you have some time to test it out? Hold on, found a bug. ill update shortly.

mathpaquette commented 3 years ago

@BunkerCoder have a look https://github.com/mathpaquette/IQFeed.CSharpApiClient/commit/21cd24b2294fb5ce89c40d5b7b66e807731501c0

BunkerCoder commented 3 years ago

@mathpaquette I'm bogged down today on a client fire, and everyone is on vacation! I'll try to look later today. Thanks for jumping on this. I will too as soon as I can.

BunkerCoder commented 3 years ago

@mathpaquette I ran "git checkout 21cd24b", then built, then re-targeted my throwing app to use the local library. Throwing app still throws with "E,Too many simultaneous history requests." Thrown from BaseLookupFacade.cs:line 61 as expected. Will do some more digging.

mathpaquette commented 3 years ago

@mathpaquette I ran "git checkout 21cd24b", then built, then re-targeted my throwing app to use the local library. Throwing app still throws with "E,Too many simultaneous history requests." Thrown from BaseLookupFacade.cs:line 61 as expected. Will do some more digging.

try to decrease 30 here https://github.com/mathpaquette/IQFeed.CSharpApiClient/commit/21cd24b2294fb5ce89c40d5b7b66e807731501c0#r45348806

BunkerCoder commented 3 years ago

Tried with 10. Same result.

mathpaquette commented 3 years ago

@BunkerCoder wow... Ill look at it. here is the official change from IQFeed http://forums.iqfeed.net/index.cfm?page=topic&topicID=5832 Just got the email.

Mathieu

mathpaquette commented 3 years ago

@BunkerCoder are you able to reproduce now? Im not able to get the exception.... Can you share a sample with me to reproduce it?

BunkerCoder commented 3 years ago

@mathpaquette I've started looking at it, Tried adding a stress test in ConcurrentHistoricalExample.cs, and could not get it to fail, so assuming that your solution is working, and I've got a bug in my app. Will advise results.

mathpaquette commented 3 years ago

@mathpaquette I've started looking at it, Tried adding a stress test in ConcurrentHistoricalExample.cs, and could not get it to fail, so assuming that your solution is working, and I've got a bug in my app. Will advise results.

no I dont think so. I think they simply turned it off for now. Lets see. Tell me if you notice something on your side... Thanks

BunkerCoder commented 3 years ago

Now that I look again, I think you're right. I re-tested my throwing app and it didn't throw.

mathpaquette commented 3 years ago

@BunkerCoder I think their implementation was broken and they decided to roll it back.. lets see.

BunkerCoder commented 3 years ago

Received DTN email; rate limit removed until 12-Jan, when 50 requests/sec. limit goes into effect. No word about a test server.

Maybe they will open up an API call to check the rate; multiple simultaneous apps/clients can't see the overall machine rate. At a minimum, I think that LookupRateLimiter() should be a shared instance or static instead of per-Client.

mathpaquette commented 3 years ago

50 requests/sec isssssssss crazy fast. 3000 reqs per minute... we should be good to go. right now, every class instances from the same LookupClient is using the same rate limiter. Do you have use case of having multiple LookupClient per program?

BunkerCoder commented 3 years ago

Not yet, but I'm working on one! If it becomes a problem, I'll workup a solution and submit a PR. Maybe just change 30 to 50 for now?

BunkerCoder commented 3 years ago

Strike that. Don't see a good use case for multiple LookupClient per program. But may possibly see running multiple programs, so being able to tweak each program would be nice to have. Maybe a constructor that supported: var lookupRateLimiter = new LookupRateLimiter(requestsPerSecondLimit); Happy to contribute.

mathpaquette commented 3 years ago

Yes absolutely. It’s gonna be a parameter from the factory for sure!

Thanks

Sent from my iPhone

On Dec 23, 2020, at 7:43 PM, BunkerCoder notifications@github.com wrote:



Strike that. Don't see a good use case for multiple LookupClient per program. But may possibly see running multiple programs, so being able to tweak each program would be nice to have. Maybe a constructor that supported: var lookupRateLimiter = new LookupRateLimiter(requestsPerSecondLimit); Happy to contribute.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/mathpaquette/IQFeed.CSharpApiClient/issues/90#issuecomment-750652928, or unsubscribehttps://github.com/notifications/unsubscribe-auth/ABBICN6YXK6VSSPVJRXUGYDSWKFCRANCNFSM4VE2JTMQ.

BunkerCoder commented 3 years ago

@mathpaquette Silly of me to have even mentioned it. Have I said Thank You lately?

mathpaquette commented 3 years ago

@BunkerCoder your involvement is significantly more meaningful than few words :)

Nucs commented 3 years ago

Too bad I missed this, What a mess this change has created.

Here is the official email from 23 Dec 2020:

Title: IQFeed NOTICE: IQFeed history request rate limiting change IQFeed history request rate limiting change. Last weekend, IQFeed released a server-side update to our historical data servers which changed the method that we use to rate limit requests. This change was necessary to protect our servers from rogue software and to help standardize and improve the performance of our history servers for all customers. This change ONLY applies to our history servers.

Previously, customers were allowed to have 15 simultaneous requests active at any given time. While this was a good rate limiting method when it was originally developed, advances in machine hardware and reduced latency to our servers over the internet had effectively removed this limit entirely. Customers were able to make nearly unlimited requests and that was never the intent for our service. This method of request rate limiting was confusing for new customers. It was dependent upon latency to our servers so each customer had a different limit that they could achieve. This was also never intended, and led to mixed results and customer experiences. For years we were able to operate this way using an informal "honor system", by closely monitoring our server performance and reaching out to customers to adjust their software as needed. However, this approach is no longer feasible.

The new method is a straightforward "requests per second" rate limiting method. It is a bucket limit system similar to the Linux kernel's iptables hashlimit. As a result, you start with a full bucket of request credits. Each request consumes 1 credit and credits are re-allocated every few milliseconds unless you are already at max. If you send a request when you are out of requests, you will get an error stating too many requests (the error message is the same as before). This new method allows all customers to have the same limit. Also, customers can monitor the limit themselves, and know their real limit without trial and error

With all that said, the new limit for requests is 50/second (with one new credit added each 20ms). For the vast majority of customers, this change will not affect their IQFeed performance at all. Unfortunately, for a few of you who are downloading a large symbol list as fast as possible, you will notice this change (if you haven't already). Almost all of you who fall into this category can still get the data you need, but it will take you a bit more time to make the requests. If you fall into this category, we urge you to reach out to our dev support team with a description of the data you need and the requests you are currently trying to use to download it. Our support staff will help you figure out the best way to get this data from the feed.

We would also like to offer an apology for our lack of fore-warning on this change. We understand that we should have sent this message before we rolled out the change so that you had time to analyze your needs and adjust accordingly ahead of time. We will do better in the future.

We welcome any and all feedback concerning these new features (and old features). You can send feedback, questions, or concerns, by replying to this email or contact developer support on the forums or chat.

Thanks, IQFeed Development Team

mathpaquette commented 3 years ago

guys.... spent few hours thinking about of one precise working solution... I think I got something.... Ill sharing today.

BunkerCoder commented 3 years ago

Many thanks for a very elegant, ,highly-performant, and timely-provided solution!

mathpaquette commented 3 years ago

Apparently, 50 requests per second is too high... I'm getting , ErrorMessage: Too many simultaneous history requests., MessageTrace: E,Too many simultaneous history requests., !ENDMSG!,

mathpaquette commented 3 years ago

Apparently 50 requests per second works well but you have to limit to 8 concurrent clients.

BunkerCoder commented 3 years ago

I got the error with 6 concurrent clients @ 50/sec. a couple of times this week. Instead of fighting it, I dialed back to 45/sec., and haven't seen the error since.