search

softwarespartan / IB4m

Interactive Brokers API for Matlab
GNU General Public License v2.0
62 stars 21 forks source link

Just asking for help: Receiving limited and incorrect historical data #13

Open mughalgit opened 6 years ago

mughalgit commented 6 years ago

Dear Abel,

Firstly, my sincere thanks for putting together this tool and for the online documentation.

I want to be able to request recent historical data for FaceBook. To do this I followed your example (http://softwarespartan.github.io/IB4m/docs/html/HistoricalDataExample.html) and this works fine. In fact I can compare the graph generated for the close value of FB (for the year of 2014) with the online graph at https://finance.yahoo.com/ and see that they agree with each other.

However, as a next step I tried adapting your example to get minute-by-minute data for Facebook's close value for the previous day. This is where my problem starts. Here is what I did:

=========================================================================== cd /Users/adilmughal/Applications/IB4m; addpath(path,pwd); addpath(path,fullfile(pwd,'docs')); javaaddpath(fullfile(pwd,'Jar','TWS.jar')); session = TWS.Session.getInstance(); session.eClientSocket.eConnect('127.0.0.1',7497,0); added interface method: TWSNotification notification listener has been added Server Version:76 TWS Time at connection:20171020 10:32:13 Greenwich Mean Time -1 2104 Market data farm connection is OK:hfarm

-1 2104 Market data farm connection is OK:eufarm

-1 2104 Market data farm connection is OK:jfarm

-1 2104 Market data farm connection is OK:usfuture

-1 2104 Market data farm connection is OK:cashfarm

-1 2104 Market data farm connection is OK:usfarm.us

-1 2104 Market data farm connection is OK:usfarm

-1 2106 HMDS data farm connection is OK:ilhmds

-1 2106 HMDS data farm connection is OK:fundfarm

-1 2106 HMDS data farm connection is OK:ushmds

-1 2106 HMDS data farm connection is OK:euhmds

[buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA); contract = com.tws.ContractFactory.GenericStockContract('FB');

session.eClientSocket.reqHistoricalData(1070001,contract,'20171019 16:00:00', '1 D', '1 min', 'TRADES',1,1,[]); pause(0.5);

hde = buf.get(); bars = collection2cell(hde.data); numel(bars)

ans =

90

numel(bars) == hde.data.size()

ans =

logical

1

for i = 1:min(numel(bars),10); disp(bars{i}); end 1070001 20171019 15:48:00 173.47 173.50 173.43 173.48 180 132

1070001 20171019 15:27:00 173.50 173.52 173.35 173.49 308 183

1070001 20171019 15:25:00 173.52 173.58 173.46 173.49 301 195

1070001 20171019 15:21:00 173.74 173.78 173.46 173.66 595 319

1070001 20171019 15:55:00 173.82 173.95 173.81 173.93 396 188

1070001 20171019 15:50:00 173.49 173.73 173.47 173.70 440 209

1070001 20171019 15:37:00 173.33 173.41 173.31 173.39 388 191

1070001 20171019 15:52:00 173.69 173.76 173.66 173.71 251 134

1070001 20171019 15:18:00 173.54 173.73 173.52 173.68 433 210

1070001 20171019 14:56:00 173.10 173.13 172.85 172.88 694 375

===========================================================================

Where in "session.eClientSocket.reqHistoricalData" the time "20171019 16:00:00" represents yesterday's date at 4pm.

My first problem is that I don't understand why I only get 90 bars worth of data (i.e. 1.5 hours worth) - I would have expected to get the entire day's worth.

My second problem is that the data seems to be wrong. So for example the first bar of data is:

1070001 20171019 15:48:00 173.47 173.50 173.43 173.48 180 132

However, according to https://finance.yahoo.com/ the close value of Facebook at 15:48:00 is 174.35

And in fact if I compare the graph from Yahoo:

safariscreensnapz002

with the graph from Matlab/IB4M:

matlabscreensnapz001

There is no correspondence between them. I guess I must be doing something fundamentally wrong but I don't understand.

Thank you in advance for your help

A

softwarespartan commented 6 years ago

Hi Adil

Thanks for reaching out.

Looks like everything is functional/correct within your IB4m setup.

That means, it is just an issues getting the historical data call/query to give you back the data you want.

First, please review IB docs for historical data bars and limitations: 

https://interactivebrokers.github.io/tws-api/historical_bars.html#gsc.tab=0 <https://interactivebrokers.github.io/tws-api/historical_bars.html#gsc.tab=0>

https://interactivebrokers.github.io/tws-api/historical_limitations.html#gsc.tab=0 <https://interactivebrokers.github.io/tws-api/historical_limitations.html#gsc.tab=0>

For regular stock contracts the requests are fairly simply once you get comfortable with the general query “language”.

Also note that when making historical data requests for Futures and Options the contract configuration takes a bit more work.

Here is a function that I use often to make simple historical data queries for stocks.

function [s,bars] = historicDataQuery(contract,dt,config)

% some randome number for unique request id r = round(rand*10000);

% get TWS session session = TWS.Session.getInstance();

% connect the session to TWS running instance session.eClientSocket.eConnect('127.0.0.1',7496,0);

% create local buffer for historical data events [buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA);

% loop over number of requests for i = 1:config.N

% create unique id
requestId = r+100000+i;

% compute the end time of the request
%endDateTime = datestr(dt-config.delta*(i-1),'yyyymmdd HH:MM:SS');
endDateTime = datestr(dt-config.delta*(i-1),'yyyymmdd 16:00:00');

% make the request for data
session.eClientSocket.reqHistoricalData(  ...
    requestId         ,...
    contract          ,...
    endDateTime       ,...
    config.duration   ,...
    config.barsize    ,...
    config.whatToShow ,...
    config.useRTH     ,...
    1                 ,...
    []                 ...
);

pause(0.5);

end

% wait around for the data pause(3);

% init hde hde = cell(1,buf.size);

% Retreive the historical data event from the buffer for i = 1:buf.size; hde{i} = buf.get(); buf.remove(); end

% Convert the HashSet into more native cell array of bars bars = {}; for i = 1:numel(hde); bars = [bars; collection2cell(hde{i}.data)]; end %#ok

% Extract close and high low ave info from each bar` close = cellfun(@(b)b.close ,bars);
vol = cellfun(@(b)b.volume,bars);
high = cellfun(@(b)b.high ,bars);
low = cellfun(@(b)b.low ,bars);
open = cellfun(@(b)b.open ,bars); wap = cellfun(@(b)b.wap ,bars); count = cellfun(@(b)b.count ,bars); hla = cellfun(@(b)(b.high+b.low)/2,bars);

% Convert the string time of the bars to matlab datenum dt = datenum(cellfun(@(b)char(b.dtstr),bars,'UniformOutput',false),'yyyymmdd HH:MM:SS');

% The bars might not be in order once extracted from the HashSet [~,indx] = sort(dt);

% sort time stamps and get unique dt = dt(indx); [~,uix] = unique(dt);

% sort and assign s.dt = dt (uix); close = close(indx); s.close = close(uix); hla = hla (indx); s.hla = hla (uix); high = high (indx); s.high = high (uix);
low = low (indx); s.low = low (uix);
open = open (indx); s.open = open (uix); vol = vol (indx); s.volume = vol (uix); wap = wap (indx); s.wap = wap (uix); count = count(indx); s.count = count(uix);

% sort the bars bars = bars(indx); bars = bars(uix);

% clean up listener handle cellfun(@(l)delete(l),lh);

Once you have this function it is easy to make a historical data request. For example:

% date and time for yesterday (-1) dt = now -1;

% lets have a look at some symbol symbol = 'FB';

% create generic contract specification for this symbol contract = com.tws.ContractFactory.GenericStockContract(symbol);

% configure the data request config.symbol = symbol ; config.duration = '1 D' ; config.barsize = '1 min' ; config.whatToShow = 'TRADES'; config.useRTH = 1 ; config.delta = 1 ; config.N = 1 ;

% make the data request for contract [dat,bars] = historicDataQuery(contract,dt,config);

Which returns

dat

dat =

struct with fields:

    dt: [390×1 double]
 close: [390×1 double]
   hla: [390×1 double]
  high: [390×1 double]
   low: [390×1 double]
  open: [390×1 double]
volume: [390×1 double]
   wap: [390×1 double]
 count: [390×1 double]

When plotting these data and comparing with TWS, the data returned from historical API query and TWS match exactly (see attached plot).

Similarly, if you wanted 3 years of '1 day’ data/bars for ‘FB’ then you would use the configuration:

% configure the data request config.symbol = symbol ; config.duration = '1 Y' ; config.barsize = '1 day' ; config.whatToShow = 'TRADES'; config.useRTH = 1 ; config.delta = 365 ; config.N = 3 ;

Do not hesitate to reach out with additional questions.

Cheers -abel

On Oct 20, 2017, at 6:17 AM, mughalgit notifications@github.com wrote:

Dear Abel,

Firstly, my sincere thanks for putting together this tool and for the online documentation.

I want to be able to request recent historical data for FaceBook. To do this I followed your example (http://softwarespartan.github.io/IB4m/docs/html/HistoricalDataExample.html http://softwarespartan.github.io/IB4m/docs/html/HistoricalDataExample.html) and this works fine. In fact I can compare the graph generated for the close value of FB (for the year of 2014) with the online graph at https://finance.yahoo.com/ https://finance.yahoo.com/ and see that they agree with each other.

However, as a next step I tried adapting your example to get minute-by-minute data for Facebook's close value for the previous day. This is where my problem starts. Here is what I did:

=========================================================================== cd /Users/adilmughal/Applications/IB4m; addpath(path,pwd); addpath(path,fullfile(pwd,'docs')); javaaddpath(fullfile(pwd,'Jar','TWS.jar')); session = TWS.Session.getInstance(); session.eClientSocket.eConnect('127.0.0.1',7497,0); added interface method: TWSNotification notification listener has been added Server Version:76 TWS Time at connection:20171020 10:32:13 Greenwich Mean Time -1 2104 Market data farm connection is OK:hfarm

-1 2104 Market data farm connection is OK:eufarm

-1 2104 Market data farm connection is OK:jfarm

-1 2104 Market data farm connection is OK:usfuture

-1 2104 Market data farm connection is OK:cashfarm

-1 2104 Market data farm connection is OK:usfarm.us

-1 2104 Market data farm connection is OK:usfarm

-1 2106 HMDS data farm connection is OK:ilhmds

-1 2106 HMDS data farm connection is OK:fundfarm

-1 2106 HMDS data farm connection is OK:ushmds

-1 2106 HMDS data farm connection is OK:euhmds

[buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA); contract = com.tws.ContractFactory.GenericStockContract('FB');

session.eClientSocket.reqHistoricalData(1070001,contract,'20171019 16:00:00', '1 D', '1 min', 'TRADES',1,1,[]); pause(0.5);

hde = buf.get(); bars = collection2cell(hde.data); numel(bars)

ans =

90 numel(bars) == hde.data.size()

ans =

logical

1

for i = 1:min(numel(bars),10); disp(bars{i}); end 1070001 20171019 15:48:00 173.47 173.50 173.43 173.48 180 132

1070001 20171019 15:27:00 173.50 173.52 173.35 173.49 308 183

1070001 20171019 15:25:00 173.52 173.58 173.46 173.49 301 195

1070001 20171019 15:21:00 173.74 173.78 173.46 173.66 595 319

1070001 20171019 15:55:00 173.82 173.95 173.81 173.93 396 188

1070001 20171019 15:50:00 173.49 173.73 173.47 173.70 440 209

1070001 20171019 15:37:00 173.33 173.41 173.31 173.39 388 191

1070001 20171019 15:52:00 173.69 173.76 173.66 173.71 251 134

1070001 20171019 15:18:00 173.54 173.73 173.52 173.68 433 210

1070001 20171019 14:56:00 173.10 173.13 172.85 172.88 694 375

===========================================================================

Where in "session.eClientSocket.reqHistoricalData" the time "20171019 16:00:00" represents yesterday's date at 4pm.

My first problem is that I don't understand why I only get 90 bars worth of data (i.e. 1.5 hours worth) - I would have expected to get the entire day's worth.

My second problem is that the data seems to be wrong. So for example the first bar of data is:

1070001 20171019 15:48:00 173.47 173.50 173.43 173.48 180 132

However, according to https://finance.yahoo.com/ https://finance.yahoo.com/ the close value of Facebook at 15:48:00 is 174.35

And in fact if I compare the graph from Yahoo:

https://user-images.githubusercontent.com/32955429/31816421-2a2fb61e-b588-11e7-8c5a-3881373f98e9.jpg with the graph from Matlab/IB4M:

https://user-images.githubusercontent.com/32955429/31816217-605eeb70-b587-11e7-86ce-a8eee4c50245.jpg There is no correspondence between them. I guess I must be doing something fundamentally wrong but I don't understand.

Thank you in advance for your help

A

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/13, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1O8irdMwLSTi0Z1m_h7HDjrnNRtPks5suHNMgaJpZM4QAd-i.

mughalgit commented 6 years ago

Dear Abel,

Firstly - many sincere thanks for replying. Your function "historicDataQuery" works perfectly.

In fact we managed to find the source of the error - I am using GMT (since I am in the UK) and the 16:00 hours is 11:00 in New York - and so the reason I was only getting 90 minutes of data is because the exchange had only been open since 09:30. Once I understood that everything else was fine.

Once again thank you for your assistance - much appreciated.