Enhancement: suport for setting "Timeout to send bulk data to API"

[gnzsnz]

Thanks for the great work on IBC.

I'm using it extensively on a docker image with IB gateway and IBC.

I recently realized that IB gateway and TWS have different default settings for "Timeout to send bulk data to API", IB gateway uses 30 seconds and TWS 300 (at least that's what i have)

It would be great if we could control this setting through IBC's config.ini.

Hope it makes sense and thanks again for the great work on IBC

[rlktradewright]

I just did a test and the out-of-the-box default for this setting for both TWS and Gateway appearas to be 30 secs. So I don't know why you have 300 secs in your TWS: maybe your settings are stored on the server, so even though you may be starting a brand new instance of TWS, that setting is being overriden by the stored settings?

I don't really want to implement this. I've never found any use for it. I'm not even convinced that IB have actually implemented it properly: the various levels of buffering involved in the path from TWs/Gateway to API socket reading means that there's a very large amount of data that needs to be sent before TWS/Gateway will receive any kind of error. In fact, I really don't understand what it's supposed to mean. I've often had debugging sessions where my API client hasn't read the socket for ages, and TWS (or maybe the Java runtime) is writing messges to the logfile saying things like 'The buffer has grown to 800 KBytes", but the API is not disconnected.

So feel free to try to convince me that there is benefit in being able to configure this...

[gnzsnz]

the value in the documentation is 30 as well, https://www.ibkrguides.com/tws/usersguidebook/configuretws/apisettings.htm?Highlight=timeout . I might have changed it manually.

In any case my rational for this enhancement is to be able to increase the timeout through a parameter in config.ini. This is mainly for a headless IB Gateway. The case I have come up with is to request large amounts of high frequency historical data. it works well if I request it one after the other (for example SPX, historical volatility and VIX 5min bars for the last 2 years) however this is slow, so i run my requests asynchronously which takes less time in total but triggers the timeout limit

If this could be available as an option, then there is no need to manually change the value manually.

Another option could be to save IB gateway settings in a container volume. But i haven't find a way separate configuration, which i want to save, from the application, which i want to refresh with every new container start-up.

[rlktradewright]

Sorry for the delay in responding.

My view of this timeout is that if it triggers, there is potentially something badly wrong with the performance of the API client. It really ought to be possible for the client to read data from the socket fast enough to prevent this happening. The fact that it happens in the scenario you describe indicates to me that there is something you need to investigate and improve.

I ran my downloader last night to download 2 years-worth of 5-minute data for the first 9 stocks in my S&P 500 list. The downloader does everything asynchronously, making up to 20 concurrent contract details requests and up to 15 concurrent historical data requests. I set it to write the data to text files.

Here is the logfile from the run, which shows the results:

Log file

The first download starts at 00:14:12.061, and everything finishes at 00:42:15.054. That's 28mins to download 352,134 bars, so about 209 bars per second on average. Each bar is 60-70 bytes of network data, so that's about 24 MBytes of data in total, insignificant over that period. If I repeat the run with a larger or smaller number of stocks, the throughput is about the same.

So it seems to me that something about your client program is simply not keeping up. If you're doing a lot of processing per bar, perhaps it would be better to simply read the bars from the API into some sort of list, and then process them on another thread or with async tasks. You really don't ever want the API tied up for 30 seconds or more, you could be holding up all sorts of important messages, such as order status and executions (though admittedly a program that places orders is perhaps unlikely to be doing large historical data requests simultaneously).

So I don't think you've convinced me that this is a worthwhile enhancement: it's more likely to just paper over cracks that really need to be filled.

[gnzsnz]

thanks for your detailed and factual answer.

sorry for my late reply, but i could not spare some time to do my research.

I found the issue, and indeed it was on the client side. I'm using ib_insync, which works great. and i found that it was timing out my long running requests. so I'm able now to replicate your results by increasing this timeout.

I did find as well to address the other potential solution that I mentioned on my previous comment. this is to find a way to save settings in a separate directory, so I could save then on a docker volume. and i could not find a way to do it on ib gateway, but i realized that actually IBC can do it (so there must be a way through IB gateway, but i could not find in). This will allow me to save settings across containers which gives a lot of flexibility.

thanks for your help