Python async historical requests

[quazzuk]

Hi

Firstly thank you for your excellent library, it's been a godsend!

I'm currently using the library from python to download historical data, using pythonnet as per your examples. I've managed to achieve a certain amount of parallelism using multiple connections (up to 6 seems to work) and a multiprocessing pool. However, I've not found a way to use the async variants (GetHistory*Async) on the HistoricalFacade as I believe this would require some interop between python asyncio and C# tasks which I don't think is supported according to this .

Are you aware of any solutions to this? I would really like to be able to get 50 requests/sec inflight instead of being limited to 6. Downloading options data takes forever... ;-)

Thanks Andy

[mathpaquette]

hey @quazzuk first of, I'm glad to see that I'm helping others to reach their goals. :) As far as I know, its not possible to use C# async methods from python. This is pretty much why I added these sync commands along the way. BUT something you can try out is to wrap these sync methods from C# and make them async with Python using asyncio lib. I've done that recently and was pretty successful with it.

Have a look to this: https://stackoverflow.com/a/22414756/2228237

[quazzuk]

Thanks for the response.

Ok, yes I'm using loop.run_in_executor but seemed to remember hitting the "Too many simultaneous requests" limit. Perhaps that was prior to your rate throttling functionality being added.

Let me check...

[mathpaquette]

you might have to play with requestsPerSecond parameter (https://github.com/mathpaquette/IQFeed.CSharpApiClient/blob/master/src/IQFeed.CSharpApiClient/Lookup/LookupClientFactory.cs#L23) even I unit tested it carefully, apparently on IQFeed's side, doesn't work as promised.

For me, 8 concurrent clients works well with 50 requests per second but if I go above 8 clients I need to reduce it in order to avoid the limit by IQfeed.

[quazzuk]

Thanks yes. I think I originally wrote the code around Christmas when IQFeed introduced the limits and spent quite a lot of time debugging before googling and finding it had just been introduced (unannounced) on the IQ side, doh!

One separate question (apologies for it being slightly off topic) but I was curious how you perform efficient incremental updates of historical datasets? Does the IQ API only support per symbol requests or is it possible to request data spanning multiple symbols in a single call i.e. give me a single data point for each symbol within the specified date range based on security_type or exchange?

[mathpaquette]

Cool. There's no such capability. It's symbol based only. Can I close the issue?

[quazzuk]

Yes, please do. Thanks