Event buffer from TWS.Events.HISTORICALDATA only records the reqHistoricalData query that returns first and ignores all subsequent ones.

[ecpgieicg]

Hi Abel,

As always, thank you for creating and sharing the package.

I am running into the issue that the event buffer for reqHistoricalData only records the query that returns first and ignores all subsequent queries of a different TickerID.

I tried re-initializing the buffer. Sure enough it works for a new query but again a single one only. With this I run into trouble whenever the IB server response is choppy. For me at least, it often freezes and minutes later return data in bulk for all previous queries -- although not necessarily in order -- even if I observe the old Historical Data Limitations

Is there a way to get the event buffer to cache all incoming messages until a session is terminated? Or is it something else I am missing? I am a novice when it comes to programming. Any help would be appreciated.

[softwarespartan]

Thanks for reaching out.

This is strange behavior, indeed.

Do you have a simple test script you can share that reproduces the issue?

Also extra details like ‘ver’ and TWS version, operating system, etc would also be helpful

Cheers

Sent from my iPhone

On Jun 21, 2019, at 6:40 PM, ecpgieicg notifications@github.com wrote:

Hi Abel,

As always, thank you for creating and sharing the package.

I am running into the issue that the event buffer for reqHistoricalData only records the query that returns first and ignores all subsequent queries.

I tried re-initializing the buffer. Sure enough it works for a new query but again a single one only. With this I run into trouble whenever the IB server response is choppy. For me at least, it sometimes freezes and minutes later return data in bulk for all previous queries -- although not necessarily in order.

Is there a way to get the event buffer to cache all incoming messages until a session is terminated? Or is it something else I am missing? I am a novice when it comes to programming. Any help would be appreciated.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[ecpgieicg]

Certainly.

First I make sure the following is run while current folder in Matlab is set to the IB4m package folder.

javaaddpath(fullfile(pwd,'Jar','TWS973.jar'));

Afterwards, I am able to initialize the connection successfully and use reqContractDetails to query contract info.

session = TWS.Session.getInstance();
session.eClientSocket.eConnect('127.0.0.1',7497,2);

[buf,lh] = TWS.initBufferForEvent(TWS.Events.CONTRACTDETAILS);
contract = com.ib.client.Contract();
contract.symbol("AMD");contract.secType("OPT");contract.exchange("SMART");contract.currency("USD");contract.lastTradeDateOrContractMonth("20190719");contract.strike(30);contract.right("Call");
session.eClientSocket.reqContractDetails(0,contract); pause(0.5);

The customer support at IB strongly recommends always including conid. So I've never tried querying without it.

contract.conid(buf.get().data.iterator().next().contractDetails.conid)

But the followings should cause problems.

[buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA);
session.eClientSocket.reqHistoricalData(105,contract,'','1 W','1 min','BID',1,1,false,[]); pause(0.5);
session.eClientSocket.reqHistoricalData(106,contract,'','1 D','1 min','BID',1,1,false,[]); pause(0.5);
session.eClientSocket.reqHistoricalData(102,contract,'','1 W','3 mins','TRADES',1,1,false,[]); pause(0.5);

At this point, I would expect buf.get().data to only collect returns for one of the 3 queries, whichever returns first. We can tell by looking at the reqId of the individual data entries and by the fact that number of returned entries remain unchanged as more queries are made.

buf.get().data.iterator().next().reqId,
buf.get().data.size,

If you don't see the same issue, maybe delayed response from the server is required for seeing the glitch. Querying monthly/1 min candles usually result in delay for option contracts. e.g. instead of the above 3, query the following 4,

[buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA);
session.eClientSocket.reqHistoricalData(107,contract,'','1 M','1 min','BID',1,1,false,[]); pause(0.5);
session.eClientSocket.reqHistoricalData(105,contract,'','1 W','1 min','BID',1,1,false,[]); pause(0.5);
session.eClientSocket.reqHistoricalData(106,contract,'','1 D','1 min','BID',1,1,false,[]); pause(0.5);
session.eClientSocket.reqHistoricalData(102,contract,'','1 W','3 mins','TRADES',1,1,false,[]); pause(0.5);

Although I do see the issue repeated without experiencing throttle by IB server.

[ecpgieicg]

OS: Windows 7 Matlab version: 2018a API version: 9.76 TWS version: should be 9.73, I am not sure how to find it..

[softwarespartan]

Ok great. I’ll try to test out over the weekend and let you know what I find

On Fri, Jun 21, 2019 at 7:37 PM ecpgieicg notifications@github.com wrote:

OS: Windows 7 Matlab version: 2018a API version: 9.76 TWS version: should be 9.73, will come back after I check. (Not sure how you can check without restarting. I am downloading option data for Jun 21 expiry atm. Need to do that before they get removed.)

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/54?email_source=notifications&email_token=ABC2VVHF3DHYW3T772QM7OLP3VQ4JA5CNFSM4H2VIHSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJ2MZY#issuecomment-504604263, or mute the thread https://github.com/notifications/unsubscribe-auth/ABC2VVES6JHLFFNV5V5A4MDP3VQ4JANCNFSM4H2VIHSA .

[ecpgieicg]

Sounds good

I changed the contract line to point to Jul 19 expiry instead since Jun option data will be removed over the weekend.

[Despair2000]

Did you try to have a longer pauses between the requests, not just 0.5s? See what happens if you wait 5-10s between the requests.

[ecpgieicg]

Yes but it was for an unrelated purpose. Why do you suggest having longer pauses?

[Despair2000]

I experienced several times that a problem could be solved by simply waiting a few seconds (IB support pointed this out to me). Besides I run a program that loads historical data for more than 80 stocks on start up and this works fine. I have the following code in my function that fetches the data (you see the while-loop to wait for incoming data):

try session.eClientSocket.reqHistoricalData(10000+reqID,contract,endTime,duration,barSize,dataType,regularOnly,true,false,[]); catch disp('A problem occurred when I tried to download historical data for ',contract.symbol,'!') data = []; return end

i = 0; while buf.isEmpty && i < 100 pause(0.5) i = i + 1; end

if i == 100 disp('Time out! - A problem occurred when I tried to download historical data!') data = []; return end

[ecpgieicg]

Below is a helper function I use.

    function read_buffer_bars
        try
            bars = [bars;collection2cell(buf.get().data)];
        catch
            pause(1);
            waitTime=waitTime+1;
            if waitTime >= 10
                contractType=char(contractType);
                disp(['Waiting for server response. Currently requesting ',char(contract.symbol),' ',num2str(strikePrice),' ',contractType,' ',char(endTimeStr),' ',char(Length.str),' ',char(tickerSize),' ',char(whatToShow)])
                waitTime=0;
            end
            read_buffer_bars;
        end
    end

As you can see, my wait is actually longer than your suggestion.

In any case, pausing for server response is irrelevant to the problem at hand. The problem is (successful) returns (from IB server) with a different reqID from the first return is not recorded in the local cache.

[Despair2000]

I see. Isn't simply reinitializing the buffer for every request an option? This is what I do for the initial data download. Afterwards I use TWS.initMarketDataBufferWithReqId to create one buffer for each reqID I subscribe to for the continuous updates.

[ecpgieicg]

Indeed, that is what I do too. I didn't know about the function TWS.initMarketDataBufferWithReqId though. So thank you for showing me.

If the buffer can take in returns from multiple reqID, however, the package would be able to make batch request to the IB server and handle asynchronous returns. The returns are asynchronous in nature regardless. So if one query takes particular long to return, there is a chance the returns will mix with the next query that returns quick (assuming some batch download of data from multiple instruments). That's why I posed the question at the beginning.

[softwarespartan]

@ecpgieicg

I am not able to reproduce any issues with multiple historical data requests.

Can you try to include endDateTime in the historical data request. Always best to have specify endDateTime and date calculations are easy in Matlab.

Sometimes, since everything is event based it’s hard to know when API call fails. The error for the API call goes to different buffer (i.e. error buf).

If you need the latest bar then use market data or realTimeBars.

-abel

On Jun 21, 2019, at 7:33 PM, ecpgieicg notifications@github.com wrote:

Certainly.

First I make sure the following is run while current folder in Matlab is set to the IB4m package folder.

javaaddpath(fullfile(pwd,'Jar','TWS973.jar')); Afterwards, I am able to initialize the connection successfully and use reqContractDetails to query contract info.

session = TWS.Session.getInstance(); session.eClientSocket.eConnect('127.0.0.1',7497,2);

[buf,lh] = TWS.initBufferForEvent(TWS.Events.CONTRACTDETAILS); contract = com.ib.client.Contract(); contract.symbol("AMD");contract.secType("OPT");contract.exchange("SMART");contract.currency("USD");contract.lastTradeDateOrContractMonth("20190621");contract.strike(30);contract.right("Call"); session.eClientSocket.reqContractDetails(0,contract); pause(0.5); The customer support at IB strongly recommends always including conid. So I've never tried querying without it.

contract.conid(buf.get().data.iterator().next().contractDetails.conid) But the followings should cause problems.

[buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA); session.eClientSocket.reqHistoricalData(105,contract,'','1 W','1 min','BID',1,1,false,[]); pause(0.5); session.eClientSocket.reqHistoricalData(106,contract,'','1 D','1 min','BID',1,1,false,[]); pause(0.5); session.eClientSocket.reqHistoricalData(102,contract,'','1 W','3 mins','TRADES',1,1,false,[]); pause(0.5); At this point, I would expect buf.get().data to only collect returns for one of the 3 queries, whichever returns first. If you don't see the same issue, maybe delayed response from the server is required for seeing the glitch. Querying monthly/1 min candles usually result in delay for option contracts. e.g. instead of the above 3, query the following 4,

[buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA); session.eClientSocket.reqHistoricalData(107,contract,'','1 M','1 min','BID',1,1,false,[]); pause(0.5); session.eClientSocket.reqHistoricalData(105,contract,'','1 W','1 min','BID',1,1,false,[]); pause(0.5); session.eClientSocket.reqHistoricalData(106,contract,'','1 D','1 min','BID',1,1,false,[]); pause(0.5); session.eClientSocket.reqHistoricalData(102,contract,'','1 W','3 mins','TRADES',1,1,false,[]); pause(0.5); — You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/54?email_source=notifications&email_token=ABC2VVCTBDK4KZ5ZV7PJMATP3VQK7A5CNFSM4H2VIHSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYJ2H2A#issuecomment-504603624, or mute the thread https://github.com/notifications/unsubscribe-auth/ABC2VVALCJKJTT3YDNRV5HDP3VQK7ANCNFSM4H2VIHSA.

[ecpgieicg]

@softwarespartan

Hi Abel, the problem still persists on my end. And I tried it on a different machine too.

Below is the whole printout in Matlab moments ago.

>> session = TWS.Session.getInstance();
added interface method: TWSNotification
notification listener has been added
>> session.eClientSocket.eConnect('127.0.0.1',7497,2);
-1 2104 Market data farm connection is OK:cafarm
-1 2104 Market data farm connection is OK:cashfarm
-1 2104 Market data farm connection is OK:hfarm
-1 2104 Market data farm connection is OK:usfuture
-1 2104 Market data farm connection is OK:jfarm
-1 2104 Market data farm connection is OK:usfarm.nj
-1 2104 Market data farm connection is OK:eufarm
-1 2104 Market data farm connection is OK:usopt
-1 2104 Market data farm connection is OK:usfarm
-1 2106 HMDS data farm connection is OK:euhmds
-1 2106 HMDS data farm connection is OK:fundfarm
-1 2106 HMDS data farm connection is OK:ushmds
>> [buf,lh] = TWS.initBufferForEvent(TWS.Events.CONTRACTDETAILS);
>> contract = com.ib.client.Contract();
>> contract.symbol("AMD");contract.secType("OPT");contract.exchange("SMART");contract.currency("USD");contract.lastTradeDateOrContractMonth("20190816");contract.strike(34);contract.right("Call");
>> session.eClientSocket.reqContractDetails(0,contract); pause(0.5);
>> contract.conid(buf.get().data.iterator().next().contractDetails.conid)
>> [buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA);
>> session.eClientSocket.reqHistoricalData(105,contract,'20190713 16:00:00','1 W','1 min','TRADES',1,1,false,[]); pause(0.5);
>> buf.get().data.size,
ans =
    1949
>> session.eClientSocket.reqHistoricalData(106,contract,'20190713 16:00:00','1 D','1 min','BID',1,1,false,[]); pause(0.5);
>> buf.get().data.size,
ans =
    1949
>> session.eClientSocket.reqHistoricalData(102,contract,'20190713 16:00:00','1 W','3 mins','TRADES',1,1,false,[]); pause(0.5);
>> buf.get().data.size,
ans =
    1949

In the end, the size of the cache should have increased twice.

For example, below is from the TWS API access log for the last request, which shows the request was made without syntax error thrown from the server and after some delay, the requested data was indeed received.

23:10:26:281 <- 20-102-370404929-AMD-OPT-20190816-34.0-Call--SMART--USD---0-20190713 16:00:00-3 mins-1 W-1-TRADES-1-0--
23:10:34:561 -> --{º17-102-20190706  16:00:00-20190713  16:00:00-650-20190708  

In Matlab, the last cache size query was made at 23:18. Confirming the new data entries are not recorded.

[softwarespartan]

Keep in mind that buf.get() does not remove the object from the buffer.

This means that all three calls below are on the same “least recent” object in the buffer.

You should try call size() on the buf itself and see how many objects are in the buffer.

The remove() method will remove the “least recent” object from the queue.

If you look at what the collection to cell helper function does, it iterates over the whole buffer using an iterator object to cycle through the objects.

Indeed, this is def not very idiomatic of Matlab but that’s how the apache circular buffers work.

The docs for circular buffer are here

https://commons.apache.org/proper/commons-collections/javadocs/api-3.2.2/org/apache/commons/collections/buffer/CircularFifoBuffer.html

You can also set up a call back to print to the cmd window that historical data was returned. This is what the TWS.Session object does at initialization with error messages. It sets up a callback to just print event.data for every error event.

% set up listener to pipe the error messages to the command window
this.errorListenerHandle = event.listener(                          ...
                                          TWS.Events.getInstance  , ...
                                          TWS.Events.ERROR        , ...
                                          @(s,e)disp(e.event.data)  ...
                                         );

There is no limit to the number of callbacks. Create as many as you like. Just call close() on the handle and the callback goes away. You can use close all if you want to delete all callbacks.

-abel

On Jul 15, 2019, at 11:19 PM, ecpgieicg notifications@github.com wrote:

@softwarespartan https://github.com/softwarespartan Hi Abel, the problem still persists on my end. And I tried it on a different machine too.

Below is the whole printout in Matlab moments ago.

session = TWS.Session.getInstance(); added interface method: TWSNotification notification listener has been added session.eClientSocket.eConnect('127.0.0.1',7497,2); -1 2104 Market data farm connection is OK:cafarm -1 2104 Market data farm connection is OK:cashfarm -1 2104 Market data farm connection is OK:hfarm -1 2104 Market data farm connection is OK:usfuture -1 2104 Market data farm connection is OK:jfarm -1 2104 Market data farm connection is OK:usfarm.nj -1 2104 Market data farm connection is OK:eufarm -1 2104 Market data farm connection is OK:usopt -1 2104 Market data farm connection is OK:usfarm -1 2106 HMDS data farm connection is OK:euhmds -1 2106 HMDS data farm connection is OK:fundfarm -1 2106 HMDS data farm connection is OK:ushmds [buf,lh] = TWS.initBufferForEvent(TWS.Events.CONTRACTDETAILS); contract = com.ib.client.Contract(); contract.symbol("AMD");contract.secType("OPT");contract.exchange("SMART");contract.currency("USD");contract.lastTradeDateOrContractMonth("20190816");contract.strike(34);contract.right("Call"); contract.conid(buf.get().data.iterator().next().contractDetails.conid) [buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA); session.eClientSocket.reqHistoricalData(105,contract,'20190713 16:00:00','1 W','1 min','TRADES',1,1,false,[]); pause(0.5); buf.get().data.size, ans = 1949 session.eClientSocket.reqHistoricalData(106,contract,'20190713 16:00:00','1 D','1 min','BID',1,1,false,[]); pause(0.5); buf.get().data.size, ans = 1949 session.eClientSocket.reqHistoricalData(102,contract,'20190713 16:00:00','1 W','3 mins','TRADES',1,1,false,[]); pause(0.5); buf.get().data.size, ans = 1949 In the end, the size of the cache should have increased twice.

For example, below is from the TWS API access log for the last request, which shows the request was made without syntax error thrown from the server and after some delay, the requested data was indeed received.

23:10:26:281 <- 20-102-370404929-AMD-OPT-20190816-34.0-Call--SMART--USD---0-20190713 16:00:00-3 mins-1 W-1-TRADES-1-0-- 23:10:34:561 -> --{º17-102-20190706 16:00:00-20190713 16:00:00-650-20190708
In Matlab, the last cache size query was made at 23:18. Confirming the new data entries are not recorded.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/54?email_source=notifications&email_token=ABC2VVACTG25YO2PZWFPEPLP7U45DA5CNFSM4H2VIHSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ7SHQQ#issuecomment-511648706, or mute the thread https://github.com/notifications/unsubscribe-auth/ABC2VVG362RUJ2WWTUW65QDP7U45DANCNFSM4H2VIHSA.

[ecpgieicg]

I see. Thank you for the clarifications.

I didn't know buf is a circularfifobuffer. I actually didn't know what circularfifobuffer is.

I can't seem to iterate through the buffer using buf.iterator().next(), which would always return data from the earliest returned query. Is buf.remove() required to access the data?

[ecpgieicg]

Nevermind, I can iterate through now. I didn't know .iterator() creates a new object. I thought buf.iterator would refer to the same object every time.

[ecpgieicg]

@softwarespartan Hi Abel, I am trying to call close() as I am doing some testing. Do you mean lh{1}.close()? Thanks.

[softwarespartan]

You can read up on events and listeners in MATLAB here

https://www.mathworks.com/help/matlab/ref/handle.listener.html

and here

https://www.mathworks.com/help/matlab/matlab_oop/callback-execution.html

Apologies, I had my Python hat on, rather it is delete that call on the event listener handles.

Sometimes there can just be a cell array that has bunch handles for some purpose. At the end of the function or whatever, just delete each handle in the array. Something like

for i in numel(lh); delete(lh{i}); end

[ecpgieicg]

I see. Got it. Thank you for the links.

And thank you for all the above response.