keep saving real-time data

[NewbieAlex]

Hi Abel, Thanks again for the great work here! I got a question about how to save real-time data from IB4m. My understanding is that IB4m creates a buffer for the real-time data, and it should allow saving to local disk. In my case I wanted to save each tick of the real-time data, so I wrote a while-loop in matlab. However, even though I used functions including "collection2cell(databuf))" to convert buffer data to matlab arrays, the saved data were all the same, as long as it was in the loop. It seemed like the real-time data couldn't be accessed inside the loop. On the other hand, if I call the same functions manually, it worked all right. Do you have any idea why this would happen? Thank you!

-Alex

[softwarespartan]

Hi Alex

Great to hear from you.

What do you mean by "in the loop"?

Also what do you mean by "call function manually"?

Do you have code sample?

Cheers -abel

On Jun 1, 2017, at 7:21 PM, NewbieAlex notifications@github.com wrote:

Hi Abel, Thanks again for the great work here! I got a question about how to save real-time data from IB4m. My understanding is that IB4m creates a buffer for the real-time data, and it should allow saving to local disk. In my case I wanted to save each tick of the real-time data, so I wrote a while-loop in matlab. However, even though I used functions including "collection2cell(databuf))" to convert buffer data to matlab arrays, the saved data were all the same, as long as it was in the loop. It seemed like the real-time data couldn't be accessed inside the loop. On the other hand, if I call the same functions manually, it worked all right. Do you have any idea why this would happen? Thank you!

-Alex

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[NewbieAlex]

Hi Abel, By "in the loop", I meant the while-loop in matlab, in which I called a function to unwrap the buffer to market data arrays (bid/ask, volume, etc).

My code goes like this:

data_cell = {}; keep_fetching=1; cnt = 0; while keep_fetching BidAskInfo = buffer_unwrap(databuf, metabuf); cnt = cnt + 1; data_cell{cnt, 1} = BidAskInfo; end

% "databuf" and "metabuf" are buffers from the example: %http://softwarespartan.github.io/IB4m/docs/html/MarketDataExample.html

======================

So when I call the "buffer_unwrap" function in the while-loop, it always returns the same BidAskInfo. But if I call the "buffer_unwrap" function outside the loop (eg. manually), it returns the up-to-date market price information.

Thanks, -Alex

[NewbieAlex]

Also, here's the source code for buffer_unwrap:

function BidAskInfo = buffer_unwrap(databuf, metabuf)

mktDataEvents = collection2cell(databuf); mktMetaEvents = collection2cell(metabuf);

bidSize = []; bidPrice = []; askPrice = []; askSize = []; lastSalePrice = []; lastSaleSize = []; totalVolume = [];

for i = 1:min(numel(mktDataEvents),1000)

e = mktDataEvents{i};

% make sure there's no NaN
if ~isnan(e.data.value)

    switch e.data.tickId

        case 0
            % bid size
            bidSize = [bidSize; e.data.value];
        case 1
            % bid price
            bidPrice = [bidPrice; e.data.value];
        case 2
            % ask price
            askPrice = [askPrice; e.data.value];
        case 3
            % ask size
            askSize = [askSize; e.data.value];
        case 4
            % last sale price
            lastSalePrice = [lastSalePrice; e.data.value];
        case 5
            % last sale size
            lastSaleSize = [lastSaleSize; e.data.value];
        case 8
            % total volume
            totalVolume = [totalVolume; e.data.value];
        otherwise
            % no op
    end
else
    warning('NaN in e.data.value!');     
end

end

if ~isempty(lastSalePrice) BA1 = lastSalePrice(end); else BA1 = nan; end if ~isempty(lastSaleSize) BA2 = lastSaleSize(end); else BA2 = nan; end if ~isempty(bidPrice) BA3 = bidPrice(end); else BA3 = nan; end if ~isempty(bidSize) BA4 = bidSize(end); else BA4 = nan; end if ~isempty(askPrice) BA5 = askPrice(end); else BA5 = nan; end if ~isempty(askSize) BA6 = askSize(end); else BA6 = nan; end if ~isempty(totalVolume) BA7 = totalVolume(end); else BA7 = nan; end BidAskInfo.lastPrice = BA1; BidAskInfo.lastSize = BA2; BidAskInfo.bidPrice = BA3; BidAskInfo.bidSize = BA4; BidAskInfo.askPrice = BA5; BidAskInfo.askSize = BA6; BidAskInfo.totalVol = BA7;

%% BidAskInfo.date_num = (mktMetaEvents{end}.date.getYear + 1900) 10000 + (mktMetaEvents{end}.date.getMonth+1) 100 + mktMetaEvents{end}.date.getDate; BidAskInfo.intraday_time = mktMetaEvents{end}.date.getHours 10000 + mktMetaEvents{end}.date.getMinutes 100 + mktMetaEvents{end}.date.getSeconds;

BidAskInfo.weekday = mktMetaEvents{end}.date.getDay; BidAskInfo.JavaTimeNum = mktMetaEvents{end}.date.getTime; BidAskInfo.TimezoneOffset = mktMetaEvents{end}.date.getTimezoneOffset;

end

[softwarespartan]

Hi Alex

Apologies for delay, traveling all week.

Let me have a look at this tomorrow in more detail.

What you probably want to do is have the BidAskInfo struct updated by the market data callback.

You would create one call back to update the struct and another call back to put the event in the buffer.

That way any time you access BidAskInfo it is up to date. This way you never have to call unwrap buffer.

Then when you want to save all the data from the buffer at the end of the day just call unwrap and save.

-abel

On Jun 2, 2017, at 11:06 AM, NewbieAlex notifications@github.com wrote:

Also, here's the source code for buffer_unwrap:

function BidAskInfo = buffer_unwrap(databuf, metabuf)

mktDataEvents = collection2cell(databuf); mktMetaEvents = collection2cell(metabuf);

bidSize = []; bidPrice = []; askPrice = []; askSize = []; lastSalePrice = []; lastSaleSize = []; totalVolume = [];

for i = 1:min(numel(mktDataEvents),1000)

e = mktDataEvents{i};

% make sure there's no NaN if ~isnan(e.data.value)

switch e.data.tickId

    case 0
        % bid size
        bidSize = [bidSize; e.data.value];
    case 1
        % bid price
        bidPrice = [bidPrice; e.data.value];
    case 2
        % ask price
        askPrice = [askPrice; e.data.value];
    case 3
        % ask size
        askSize = [askSize; e.data.value];
    case 4
        % last sale price
        lastSalePrice = [lastSalePrice; e.data.value];
    case 5
        % last sale size
        lastSaleSize = [lastSaleSize; e.data.value];
    case 8
        % total volume
        totalVolume = [totalVolume; e.data.value];
    otherwise
        % no op
end

else warning('NaN in e.data.value!');
end end

if ~isempty(lastSalePrice) BA1 = lastSalePrice(end); else BA1 = nan; end if ~isempty(lastSaleSize) BA2 = lastSaleSize(end); else BA2 = nan; end if ~isempty(bidPrice) BA3 = bidPrice(end); else BA3 = nan; end if ~isempty(bidSize) BA4 = bidSize(end); else BA4 = nan; end if ~isempty(askPrice) BA5 = askPrice(end); else BA5 = nan; end if ~isempty(askSize) BA6 = askSize(end); else BA6 = nan; end if ~isempty(totalVolume) BA7 = totalVolume(end); else BA7 = nan; end BidAskInfo.lastPrice = BA1; BidAskInfo.lastSize = BA2; BidAskInfo.bidPrice = BA3; BidAskInfo.bidSize = BA4; BidAskInfo.askPrice = BA5; BidAskInfo.askSize = BA6; BidAskInfo.totalVol = BA7;

%% BidAskInfo.date_num = (mktMetaEvents{end}.date.getYear + 1900) 10000 + (mktMetaEvents{end}.date.getMonth+1) 100 + mktMetaEvents{end}.date.getDate; BidAskInfo.intraday_time = mktMetaEvents{end}.date.getHours 10000 + mktMetaEvents{end}.date.getMinutes 100 + mktMetaEvents{end}.date.getSeconds;

BidAskInfo.weekday = mktMetaEvents{end}.date.getDay; BidAskInfo.JavaTimeNum = mktMetaEvents{end}.date.getTime; BidAskInfo.TimezoneOffset = mktMetaEvents{end}.date.getTimezoneOffset;

end

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-305815064, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1GyCs_sDlESiLcSjn0sV4R-3UVzLks5sACTtgaJpZM4NtqM3.

[NewbieAlex]

Hi Abel,

No worries. And thanks for getting back to me!

-Yanyu

Sent from my iPhone. Hence the brevity. --YZ.

On Jun 8, 2017, at 21:44, Abel Brown notifications@github.com wrote:

Hi Alex

Apologies for delay, traveling all week.

Let me have a look at this tomorrow in more detail.

What you probably want to do is have the BidAskInfo struct updated by the market data callback.

You would create one call back to update the struct and another call back to put the event in the buffer.

That way any time you access BidAskInfo it is up to date. This way you never have to call unwrap buffer.

Then when you want to save all the data from the buffer at the end of the day just call unwrap and save.

-abel

On Jun 2, 2017, at 11:06 AM, NewbieAlex notifications@github.com wrote:

Also, here's the source code for buffer_unwrap:

function BidAskInfo = buffer_unwrap(databuf, metabuf)

mktDataEvents = collection2cell(databuf); mktMetaEvents = collection2cell(metabuf);

bidSize = []; bidPrice = []; askPrice = []; askSize = []; lastSalePrice = []; lastSaleSize = []; totalVolume = [];

for i = 1:min(numel(mktDataEvents),1000)

e = mktDataEvents{i};

% make sure there's no NaN if ~isnan(e.data.value)

switch e.data.tickId

case 0 % bid size bidSize = [bidSize; e.data.value]; case 1 % bid price bidPrice = [bidPrice; e.data.value]; case 2 % ask price askPrice = [askPrice; e.data.value]; case 3 % ask size askSize = [askSize; e.data.value]; case 4 % last sale price lastSalePrice = [lastSalePrice; e.data.value]; case 5 % last sale size lastSaleSize = [lastSaleSize; e.data.value]; case 8 % total volume totalVolume = [totalVolume; e.data.value]; otherwise % no op end else warning('NaN in e.data.value!'); end end

if ~isempty(lastSalePrice) BA1 = lastSalePrice(end); else BA1 = nan; end if ~isempty(lastSaleSize) BA2 = lastSaleSize(end); else BA2 = nan; end if ~isempty(bidPrice) BA3 = bidPrice(end); else BA3 = nan; end if ~isempty(bidSize) BA4 = bidSize(end); else BA4 = nan; end if ~isempty(askPrice) BA5 = askPrice(end); else BA5 = nan; end if ~isempty(askSize) BA6 = askSize(end); else BA6 = nan; end if ~isempty(totalVolume) BA7 = totalVolume(end); else BA7 = nan; end BidAskInfo.lastPrice = BA1; BidAskInfo.lastSize = BA2; BidAskInfo.bidPrice = BA3; BidAskInfo.bidSize = BA4; BidAskInfo.askPrice = BA5; BidAskInfo.askSize = BA6; BidAskInfo.totalVol = BA7;

%% BidAskInfo.date_num = (mktMetaEvents{end}.date.getYear + 1900) 10000 + (mktMetaEvents{end}.date.getMonth+1) 100 + mktMetaEvents{end}.date.getDate; BidAskInfo.intraday_time = mktMetaEvents{end}.date.getHours 10000 + mktMetaEvents{end}.date.getMinutes 100 + mktMetaEvents{end}.date.getSeconds;

BidAskInfo.weekday = mktMetaEvents{end}.date.getDay; BidAskInfo.JavaTimeNum = mktMetaEvents{end}.date.getTime; BidAskInfo.TimezoneOffset = mktMetaEvents{end}.date.getTimezoneOffset;

end

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-305815064, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1GyCs_sDlESiLcSjn0sV4R-3UVzLks5sACTtgaJpZM4NtqM3.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[softwarespartan]

Hi Alex

Here is a good example of how to configure callbacks for MarketData request

This example creates an generic GUI with bid/ask price and size. The callback is configured to update GUI assets.

Let me know if this helps. We can tackle the saving issue next.

Cheers -abel

function testMkDataCallbacks()

% get TWS session instance
session = TWS.Session.getInstance();

% create a callback to print error messages to the command window
lherr  = event.listener(                         ...
                        TWS.Events.getInstance  ,...
                        TWS.Events.ERROR        ,...
                        @(s,e)disp(e.event.data) ...
                       );

% connect to TWS
session.eClientSocket.eConnect('127.0.0.1',7496,0);

contract = com.ib.client.Contract();
contract.m_symbol        = 'SPY' ;
contract.m_exchange      = 'ARCA';  
contract.m_primaryExch   = 'ARCA';
contract.m_secType       = 'STK' ;
contract.m_currency      = 'USD' ;

% list of metadata ticks
genericTickList = [                                      ...
                   '100,101,105,106,107,125,165,166,'    ...
                   '225,232,221,233,236,258, 47,291,'    ...
                   '293,294,295,318,370,370,377,377,'    ...
                   '381,384,384,387,388,391,407,411,'    ...
                   '428,439,439,456, 59,459,460,499,'    ...
                   '506,511,512,104,513,514,515,516,517' ...
                  ];

h   = figure;
hp  = uipanel('Title','Main Panel','FontSize',12,'BackgroundColor','white','Position',[.25 .1 .67 .67]);
hsp = uipanel('Parent',hp,'Title','Subpanel','FontSize',12,'Position',[.4 .1 .5 .5]);
bidpriceh = uicontrol('Parent',hsp,'Style','text','String','bid price','Position',[18    50  72 36]);
bidsizeh  = uicontrol('Parent',hsp,'Style','text','String','bid size ','Position',[18    20  72 36]);
askpriceh = uicontrol('Parent',hsp,'Style','text','String','bid price','Position',[18+72 50  72 36]);
asksizeh  = uicontrol('Parent',hsp,'Style','text','String','bid size ','Position',[18+72 20  72 36]);

% Make the UI visible.
h.Visible = 'on';

function mkDatCallback(e)

    switch e.data.tickId

        case 0
            % bid size
            set(bidsizeh,'String',sprintf('%d',e.data.value))
        case 1
            % bid price 
            set(bidpriceh,'String',sprintf('%.2f',e.data.value))
        case 2 
            % ask price 
            set(askpriceh,'String',sprintf('%.2f',e.data.value))
        case 3
            % ask size 
            set(asksizeh,'String',sprintf('%d',e.data.value))
        case 4
            % last sale price
            disp('last sale price')
        case 5
            % last sale size
            disp('last sale size')
        case 8 
            % total volume
            disp('total volume')
        otherwise
            % no op
            disp(['no switch case for this tickId: ' num2str(e.data.tickId)])
    end
end

% create a callback to update GUI
guilh  = event.listener(                              ...
                        TWS.Events.getInstance       ,...
                        TWS.Events.MARKETDATA        ,...
                        @(s,e)mkDatCallback(e.event)  ...
                       );

% create unique id for this mk data request
reqId = 0;

% request market data from TWS
session.eClientSocket.reqMktData(reqId,contract,genericTickList,false,[]); 

% wait while data is collected
pause(10);

% now that we have data, tell TWS we don't need more data
session.eClientSocket.cancelMktData(reqId);

close(h)
delete(lherr); 
delete(guilh); 

end

On Jun 8, 2017, at 10:43 PM, NewbieAlex notifications@github.com wrote:

Hi Abel,

No worries. And thanks for getting back to me!

-Yanyu

Sent from my iPhone. Hence the brevity. --YZ.

On Jun 8, 2017, at 21:44, Abel Brown notifications@github.com wrote:

Hi Alex

Apologies for delay, traveling all week.

Let me have a look at this tomorrow in more detail.

What you probably want to do is have the BidAskInfo struct updated by the market data callback.

You would create one call back to update the struct and another call back to put the event in the buffer.

That way any time you access BidAskInfo it is up to date. This way you never have to call unwrap buffer.

Then when you want to save all the data from the buffer at the end of the day just call unwrap and save.

-abel

On Jun 2, 2017, at 11:06 AM, NewbieAlex notifications@github.com wrote:

Also, here's the source code for buffer_unwrap:

function BidAskInfo = buffer_unwrap(databuf, metabuf)

mktDataEvents = collection2cell(databuf); mktMetaEvents = collection2cell(metabuf);

bidSize = []; bidPrice = []; askPrice = []; askSize = []; lastSalePrice = []; lastSaleSize = []; totalVolume = [];

for i = 1:min(numel(mktDataEvents),1000)

e = mktDataEvents{i};

% make sure there's no NaN if ~isnan(e.data.value)

switch e.data.tickId

case 0 % bid size bidSize = [bidSize; e.data.value]; case 1 % bid price bidPrice = [bidPrice; e.data.value]; case 2 % ask price askPrice = [askPrice; e.data.value]; case 3 % ask size askSize = [askSize; e.data.value]; case 4 % last sale price lastSalePrice = [lastSalePrice; e.data.value]; case 5 % last sale size lastSaleSize = [lastSaleSize; e.data.value]; case 8 % total volume totalVolume = [totalVolume; e.data.value]; otherwise % no op end else warning('NaN in e.data.value!'); end end

if ~isempty(lastSalePrice) BA1 = lastSalePrice(end); else BA1 = nan; end if ~isempty(lastSaleSize) BA2 = lastSaleSize(end); else BA2 = nan; end if ~isempty(bidPrice) BA3 = bidPrice(end); else BA3 = nan; end if ~isempty(bidSize) BA4 = bidSize(end); else BA4 = nan; end if ~isempty(askPrice) BA5 = askPrice(end); else BA5 = nan; end if ~isempty(askSize) BA6 = askSize(end); else BA6 = nan; end if ~isempty(totalVolume) BA7 = totalVolume(end); else BA7 = nan; end BidAskInfo.lastPrice = BA1; BidAskInfo.lastSize = BA2; BidAskInfo.bidPrice = BA3; BidAskInfo.bidSize = BA4; BidAskInfo.askPrice = BA5; BidAskInfo.askSize = BA6; BidAskInfo.totalVol = BA7;

%% BidAskInfo.date_num = (mktMetaEvents{end}.date.getYear + 1900) 10000 + (mktMetaEvents{end}.date.getMonth+1) 100 + mktMetaEvents{end}.date.getDate; BidAskInfo.intraday_time = mktMetaEvents{end}.date.getHours 10000 + mktMetaEvents{end}.date.getMinutes 100 + mktMetaEvents{end}.date.getSeconds;

BidAskInfo.weekday = mktMetaEvents{end}.date.getDay; BidAskInfo.JavaTimeNum = mktMetaEvents{end}.date.getTime; BidAskInfo.TimezoneOffset = mktMetaEvents{end}.date.getTimezoneOffset;

end

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-305815064, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1GyCs_sDlESiLcSjn0sV4R-3UVzLks5sACTtgaJpZM4NtqM3.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-307280272, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1LA7jPih1xTx4R06X-oaKEgOZeRkks5sCLE2gaJpZM4NtqM3.

[softwarespartan]

Hi Yanyu

Here is a short example how to save market data with a callback and a struct. Note there is no need for buffers here.

Obviously you would want to preallocate the arrays in the dat struct for performance.

Let me know if this clears things up for you or if you need anything additional.

Cheers -abel

function dat = testMkDataCallbacks2()

% get TWS session instance
session = TWS.Session.getInstance();

% create a callback to print error messages to the command window
lherr  = event.listener(                         ...
                        TWS.Events.getInstance  ,...
                        TWS.Events.ERROR        ,...
                        @(s,e)disp(e.event.data) ...
                       );

% connect to TWS
session.eClientSocket.eConnect('127.0.0.1',7496,0);

contract = com.ib.client.Contract();
contract.m_symbol        = 'SPY' ;
contract.m_exchange      = 'ARCA';  
contract.m_primaryExch   = 'ARCA';
contract.m_secType       = 'STK' ;
contract.m_currency      = 'USD' ;

% list of metadata ticks
genericTickList = [                                      ...
                   '100,101,105,106,107,125,165,166,'    ...
                   '225,232,221,233,236,258, 47,291,'    ...
                   '293,294,295,318,370,370,377,377,'    ...
                   '381,384,384,387,388,391,407,411,'    ...
                   '428,439,439,456, 59,459,460,499,'    ...
                   '506,511,512,104,513,514,515,516,517' ...
                  ];

dat = struct();
dat.bidsize       = [];
dat.bidprice      = [];
dat.asksize       = [];
dat.askprice      = [];
dat.lastsaleprice = [];
dat.lastsalesize  = [];
dat.totalvolume   = [];

function mkDatCallback(e)

    switch e.data.tickId

        case 0
            % bid size
            dat.bidsize(end+1) = e.data.value;
        case 1
            % bid price 
            dat.bidprice(end+1) = e.data.value;
        case 2 
            % ask price 
            dat.askprice(end+1) = e.data.value;
        case 3
            % ask size 
            dat.asksize(end+1) = e.data.value;
        case 4
            % last sale price
            dat.lastsaleprice(end+1) = e.data.value;
        case 5
            % last sale size
            dat.lastsalesize(end+1) = e.data.value;
        case 8 
            % total volume
            dat.totalvolume(end+1) = e.data.value;
        otherwise
            % no op
            disp(['no switch case for this tickId: ' num2str(e.data.tickId)])
    end
end

% create a callback to update gui
datlh  = event.listener(                              ...
                        TWS.Events.getInstance       ,...
                        TWS.Events.MARKETDATA        ,...
                        @(s,e)mkDatCallback(e.event)  ...
                       );

% create unique id for this mk data request
reqId = 0;

% request market data from TWS
session.eClientSocket.reqMktData(reqId,contract,genericTickList,false,[]); 

% wait while data is collected
pause(10);

% now that we have data, tell TWS we don't need more data
session.eClientSocket.cancelMktData(reqId);

delete(lherr); 
delete(datlh); 

end

On Jun 8, 2017, at 10:43 PM, NewbieAlex notifications@github.com wrote:

Hi Abel,

No worries. And thanks for getting back to me!

-Yanyu

Sent from my iPhone. Hence the brevity. --YZ.

On Jun 8, 2017, at 21:44, Abel Brown notifications@github.com wrote:

Hi Alex

Apologies for delay, traveling all week.

Let me have a look at this tomorrow in more detail.

What you probably want to do is have the BidAskInfo struct updated by the market data callback.

You would create one call back to update the struct and another call back to put the event in the buffer.

That way any time you access BidAskInfo it is up to date. This way you never have to call unwrap buffer.

Then when you want to save all the data from the buffer at the end of the day just call unwrap and save.

-abel

On Jun 2, 2017, at 11:06 AM, NewbieAlex notifications@github.com wrote:

Also, here's the source code for buffer_unwrap:

function BidAskInfo = buffer_unwrap(databuf, metabuf)

mktDataEvents = collection2cell(databuf); mktMetaEvents = collection2cell(metabuf);

bidSize = []; bidPrice = []; askPrice = []; askSize = []; lastSalePrice = []; lastSaleSize = []; totalVolume = [];

for i = 1:min(numel(mktDataEvents),1000)

e = mktDataEvents{i};

% make sure there's no NaN if ~isnan(e.data.value)

switch e.data.tickId

case 0 % bid size bidSize = [bidSize; e.data.value]; case 1 % bid price bidPrice = [bidPrice; e.data.value]; case 2 % ask price askPrice = [askPrice; e.data.value]; case 3 % ask size askSize = [askSize; e.data.value]; case 4 % last sale price lastSalePrice = [lastSalePrice; e.data.value]; case 5 % last sale size lastSaleSize = [lastSaleSize; e.data.value]; case 8 % total volume totalVolume = [totalVolume; e.data.value]; otherwise % no op end else warning('NaN in e.data.value!'); end end

if ~isempty(lastSalePrice) BA1 = lastSalePrice(end); else BA1 = nan; end if ~isempty(lastSaleSize) BA2 = lastSaleSize(end); else BA2 = nan; end if ~isempty(bidPrice) BA3 = bidPrice(end); else BA3 = nan; end if ~isempty(bidSize) BA4 = bidSize(end); else BA4 = nan; end if ~isempty(askPrice) BA5 = askPrice(end); else BA5 = nan; end if ~isempty(askSize) BA6 = askSize(end); else BA6 = nan; end if ~isempty(totalVolume) BA7 = totalVolume(end); else BA7 = nan; end BidAskInfo.lastPrice = BA1; BidAskInfo.lastSize = BA2; BidAskInfo.bidPrice = BA3; BidAskInfo.bidSize = BA4; BidAskInfo.askPrice = BA5; BidAskInfo.askSize = BA6; BidAskInfo.totalVol = BA7;

%% BidAskInfo.date_num = (mktMetaEvents{end}.date.getYear + 1900) 10000 + (mktMetaEvents{end}.date.getMonth+1) 100 + mktMetaEvents{end}.date.getDate; BidAskInfo.intraday_time = mktMetaEvents{end}.date.getHours 10000 + mktMetaEvents{end}.date.getMinutes 100 + mktMetaEvents{end}.date.getSeconds;

BidAskInfo.weekday = mktMetaEvents{end}.date.getDay; BidAskInfo.JavaTimeNum = mktMetaEvents{end}.date.getTime; BidAskInfo.TimezoneOffset = mktMetaEvents{end}.date.getTimezoneOffset;

end

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-305815064, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1GyCs_sDlESiLcSjn0sV4R-3UVzLks5sACTtgaJpZM4NtqM3.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-307280272, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1LA7jPih1xTx4R06X-oaKEgOZeRkks5sCLE2gaJpZM4NtqM3.

[NewbieAlex]

Thank you Abel! This definitely clears things up. A follow-up question, if I need to record real-time prices of two products simultaneously (say SPY and AAPL), can I do something like this:

[databuf_SPY,datalh_SPY] = TWS.initBufferForEvent(TWS.Events.MARKETDATA,100); [databuf_AAPL,datalh_AAPL] = TWS.initBufferForEvent(TWS.Events.MARKETDATA,100); session.eClientSocket.reqMktData(1,contract_SPY,genericTickList,false,[]); pause(1); session.eClientSocket.reqMktData(2,contract_AAPL,genericTickList,false,[]); pause(1);

I was assuming that the data stream of SPY would go to databuf_SPY while at the same time the AAPL data would go to databuf_AAPL. When I implemented this way, however, databuf_SPY and databuf_AAPL seemed give the same data (i.e. each of them turned out to return a mixture of SPY and AAPL prices). Do you think maybe I should've started a new session for a new product data request? Thanks in advance!

-Alex

[softwarespartan]

Hi Alex

Great question!

No need for multiple sessions.

First, read up on event listeners in MATLAB (here https://www.mathworks.com/help/matlab/matlab_oop/events-and-listeners--concepts.html).

TWS.Events just defines a set of events that anything in matlab can subscribe to. These events are plain’ol native matlab events. Just predefined since that is what internally TWS/IB4m will use to broadcast those events.

When the buffer is initialized with TWS.initBufferForEvents you see that the function registers a callback that listens to whatever event and when one arrives it just puts the events in the buffer.

Typically, the initBufferForEvent function is just used for demonstration purposes. Typically for real implementation of trading algorithms, just define whatever callbacks you need for your algorithms.

You see this in the two examples I shared for the GUI and saving/aggregating the market data. Neither use buffers, just define callbacks “on the fly” to configure how I want my program to respond to events from TWS.

From TWS perspective all data requests (both historical and market) are associated with unique requestId’s.

You have to use these ID to figure out what/which part of your program responds to which events.

For example, you could easily initialize a buffer which only responded to events with a particular reqID. Here the callback function for that buffer would check the reqID first, if the reqID for that market data event did not match the reqID of interest then do not enqueue the event. This way all objects in MATLAB can listen for events of type TWS.Events.MARKETDATA but not get confused about whose data is whose.

Does that make sense?

Now, with this in mind, looking at the code snippet you provided we can see that both databuf_SPY and databuf_APPL listen to the same TWS.Event.MARKETDATA event. Even if you have 20 different market data request active, they will all come via TWS.Event.MARKETDATA. Since the function TWS.initBufferForEvent just creates the SAME generic/anonymous callback function both buffers, both buffers get both data.

—> GOTCHA: Matlab does not allow for control logic in anonymous functions!

This means that to construct a callback with something like

if reqID == myIDofInterest; buf.queue(event); end 

you must define an actual function, explicit, nested, or otherwise.

OK, now taking all of this together, here is a function which creates buffers for market data subscriptions for a particular request id. This means that it creates buffers that listen for market data events from TWS but only enqueue them if they match a particular request ID.

function [buf,lh] = initMarketDataBufferWithReqId(bufsz,reqid)

% enforce function signature
if nargin ~= 2 ; error('usage: initMarketDataBufferWithReqId(bufsz,reqid)'); end

% define check for integer argments 
intArgCheck = @(arg)( isa(arg,'double') && arg == floor(arg) && arg >= 0 );

% enforece bufsz and reqid types and limits
if ~intArgCheck(bufsz) || ~intArgCheck(reqid); error('arg2, arg3 -- must be positive nonzero integers'); end

% create buffer for notification events 
buf = org.apache.commons.collections.buffer.CircularFifoBuffer(bufsz); 

% define nested callback function
function callback(~,e); if e.event.data.reqId == reqid; buf.add(e.event); end; end

% create listener with callback tied to reqid and buffer
lh = event.listener(                        ...
                    TWS.Events.getInstance, ...
                    TWS.Events.MARKETDATA , ...
                    @callback               ...
);   

end

Notice that the function defines a nested function as the callback for each buffer. This function is customized to only enqueue market data events with a specific reqId

To test this just create two buffers each with a different reqID and make multiple market data request at the same time.

function [bufSPY,bufQQQ] = testMkDataCallbacks3()

% create generic request ids for TWS Market Data subscriptions
reqIdSPY = 1;  reqIdQQQ = 2;

% create two market data buffers for reqid=1 and reqid=2
[bufSPY,lh1] = initMarketDataBufferWithReqId(32,reqIdSPY);
[bufQQQ,lh2] = initMarketDataBufferWithReqId(32,reqIdQQQ);

% create a callback to print error messages to the command window
lherr = event.listener(TWS.Events.getInstance, TWS.Events.ERROR, @(s,e)disp(e.event.data));

% create a contract for SPY                   % create a contract for QQQ
contractSPY = com.ib.client.Contract();       contractQQQ = com.ib.client.Contract();
contractSPY.m_symbol        = 'SPY' ;         contractQQQ.m_symbol        = 'QQQ' ;
contractSPY.m_exchange      = 'ARCA';         contractQQQ.m_exchange      = 'ARCA';  
contractSPY.m_primaryExch   = 'ARCA';         contractQQQ.m_primaryExch   = 'ARCA';
contractSPY.m_secType       = 'STK' ;         contractQQQ.m_secType       = 'STK' ;
contractSPY.m_currency      = 'USD' ;         contractQQQ.m_currency      = 'USD' ;

% list of metadata ticks
genericTickList = [                                      ...
                   '100,101,105,106,107,125,165,166,'    ...
                   '225,232,221,233,236,258, 47,291,'    ...
                   '293,294,295,318,370,370,377,377,'    ...
                   '381,384,384,387,388,391,407,411,'    ...
                   '428,439,439,456, 59,459,460,499,'    ...
                   '506,511,512,104,513,514,515,516,517' ...
                  ];

% get TWS session instance
session = TWS.Session.getInstance();

% connect to TWS
session.eClientSocket.eConnect('127.0.0.1',7496,0);

% request market data from TWS for SPY and QQQ
session.eClientSocket.reqMktData(reqIdSPY, contractSPY, genericTickList, false, []);
session.eClientSocket.reqMktData(reqIdQQQ, contractQQQ, genericTickList, false, []);

% wait around while data is collected
pause(10);

% now that we have data, tell TWS we don't need more data
session.eClientSocket.cancelMktData(reqIdSPY);
session.eClientSocket.cancelMktData(reqIdQQQ);

% make sure to disposition handles (really important when working with events)
delete(lh1);  delete(lh2);  delete(lherr);

end

When you run this

[bufSPY,bufQQQ] = testMkDataCallbacks3

You should see that both buffers collected events but that all the events in the SPY buffer have reqId =1 and all events in the bufQQQ have reqId=2.

Make sure to look through all the events in each buffer and convince yourself.

Hopefully this all makes sense.

Let me know if you have any other questions.

P.S. if you really want/need to scale market data subscriptions for a production algorithm then we can talk about TWS.MarketData.EventHandler which very efficient way in IB4m to handle huge number of objects/listeners that all have intermingled market data dependencies. To use this, listeners must subclass TWS.MarketData.EventListener and implement the process() function.

Cheers -abel

On Jun 15, 2017, at 4:47 PM, NewbieAlex notifications@github.com wrote:

Thank you Abel! This definitely clears things up. A follow-up question, if I need to record real-time prices of two products simultaneously (say SPY and AAPL), can I do something like this:

[databuf_SPY,datalh_SPY] = TWS.initBufferForEvent(TWS.Events.MARKETDATA,100); [databuf_AAPL,datalh_AAPL] = TWS.initBufferForEvent(TWS.Events.MARKETDATA,100); session.eClientSocket.reqMktData(1,contract_SPY,genericTickList,false,[]); pause(1); session.eClientSocket.reqMktData(2,contract_AAPL,genericTickList,false,[]); pause(1);

I was assuming that the data stream of SPY would go to databuf_SPY while at the same time the AAPL data would go to databuf_AAPL. When I implemented this way, however, databuf_SPY and databuf_AAPL seemed give the same data (i.e. each of them turned out to return a mixture of SPY and AAPL prices). Do you think maybe I should've started a new session for a new product data request? Thanks in advance!

-Alex

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-308861203, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1DcJqfwrtkyOUz3qHYeRo6iO2nswks5sEZhGgaJpZM4NtqM3.

[NewbieAlex]

Hi Abel,

Thank you so much for all the examples! It definitely makes more sense to me now. In addition, this might be relevant with scaling up the data subscription (which I feel very much interested): I found that if I were to define callbacks “on the fly”, I would have to wait multiple seconds (eg. 10s) before getting the prices, which is different than defining one buffer and constantly reading market data from it (can get milliseconds price information). So in the case of fast market price movement, is there a way to subscribe multiple products, and get market data at higher time resolution (eg. sub-second)? Thanks!

-Alex

[softwarespartan]

Hi Alex

No problem. Always happy to help. You mention having to wait 10s of seconds before getting the price tick. Are you sure about this? Using “on-the-fly” callbacks should be very fast. Do you have sample code I can run to test/reproduce the lag? TWS/Interactive Brokers will only issue/send a tick when something changes. So you might just be waiting for the bid/ask price to change. Obviously, test this out on something like SPY and QQQ which have tons of action. When I run those examples on my machine I get very very little lag. The final note is that Interactive Brokers consolidates the data on their end so it’s not “true” market level 1. I think at most I get like 6 updates/second. Also, if you are physically located far from the IB servers your delay will be much longer.

Anyway, if you have some code i can test send it on over and I’ll have a look into the delay you mentioned.

Cheers -abel

On Jun 19, 2017, at 4:44 PM, NewbieAlex notifications@github.com wrote:

Hi Abel,

Thank you so much for all the examples! It definitely makes more sense to me now. In addition, this might be relevant with scaling up the data subscription (which I feel very much interested): I found that if I were to define callbacks “on the fly”, I would have to wait multiple seconds (eg. 10s) before getting the prices, which is different than defining one buffer and constantly reading market data from it (can get milliseconds price information). So in the case of fast market price movement, is there a way to subscribe multiple products, and get market data at higher time resolution (eg. sub-second)? Thanks!

-Alex

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-309568313, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1M5JNJ80rYwI7HUEReChu_UroI-wks5sFt3GgaJpZM4NtqM3.

[NewbieAlex]

Hi Abel, Sure, I was using a modified version of "dat = testMkDataCallbacks2()" (code attached). In the original version there was a "pause(10)" command, which I assume was to wait for data from IB. I tried to decrease the waiting time to ~0.2s, but the returned data seemed to be less stable than a longer waiting time (would sometimes return empty values for bid/ask info). Like you said, there could be at most ~6 updates/second. Can we achieve that using functions like "testMkDataCallbacks2()", or it should be some other ways?

Thanks! -Alex

---

function dat = testMkDataCallbacks2()

% get TWS session instance
session = TWS.Session.getInstance();

% create a callback to print error messages to the command window
lherr  = event.listener(                         ...
                        TWS.Events.getInstance  ,...
                        TWS.Events.ERROR        ,...
                        @(s,e)disp(e.event.data) ...
                       );
clientId = 2;

% connect to TWS
session.eClientSocket.eConnect('127.0.0.1',7496,clientId);

contract = com.ib.client.Contract();
contract.m_symbol        = 'SPY' ;
contract.m_exchange      = 'ARCA';
contract.m_primaryExch   = 'ARCA';
contract.m_secType       = 'STK' ;
contract.m_currency      = 'USD' ;

% list of metadata ticks
genericTickList = ['225']; % get realtime auction data

dat = struct();
dat.bidsize       = [];
dat.bidprice      = [];
dat.asksize       = [];
dat.askprice      = [];
dat.lastsaleprice = [];
dat.lastsalesize  = [];
dat.totalvolume   = [];

function mkDatCallback(e)

    switch e.data.tickId

        case 0
            % bid size
            dat.bidsize(end+1) = e.data.value;
        case 1
            % bid price
            dat.bidprice(end+1) = e.data.value;
        case 2
            % ask price
            dat.askprice(end+1) = e.data.value;
        case 3
            % ask size
            dat.asksize(end+1) = e.data.value;
        case 4
            % last sale price
            dat.lastsaleprice(end+1) = e.data.value;
        case 5
            % last sale size
            dat.lastsalesize(end+1) = e.data.value;
        case 8
            % total volume
            dat.totalvolume(end+1) = e.data.value;
        otherwise
            % no op
            disp(['no switch case for this tickId: ' num2str(e.data.tickId)])
    end
end

% create a callback to update gui
datlh  = event.listener(                              ...
                        TWS.Events.getInstance       ,...
                        TWS.Events.MARKETDATA        ,...
                        @(s,e)mkDatCallback(e.event)  ...
                       );

% create unique id for this mk data request
reqId = 0;

% request market data from TWS
session.eClientSocket.reqMktData(reqId,contract,genericTickList,false,[]);

% wait while data is collected
pause(0.2);

% now that we have data, tell TWS we don't need more data
session.eClientSocket.cancelMktData(reqId);

delete(lherr);
delete(datlh);

end

---

[softwarespartan]

Hi Alex

testMktataCallbacks2 just shows how to configure callbacks on-the-fly for whatever you’re trying to do. You absolutely do not want to use “pause” when listening for market data in a real application. I just use pause there to keep the example simple and continuous flow. You definitely would need to take the examples shown in testMktataCallbacks2 and modify for your own needs. Also, you would need to pre allocate arrays as array(end+1) = value is very slow. Again, I do it in the example to keep things simple.

Maybe email me directly if you want to talk about your application more specifically. Also, make sure you read up on event handling in matlab. Event programing is not conceptually the same as procedural programing. I think you might be trying to create a procedural program when everything is event based.

Cheers -abel

On Jun 20, 2017, at 11:01 AM, NewbieAlex notifications@github.com wrote:

Hi Abel, Sure, I was using a modified version of "dat = testMkDataCallbacks2()" (code attached). In the original version there was a "pause(10)" command, which I assume was to wait for data from IB. I tried to decrease the waiting time to ~0.2s, but the returned data seemed to be less stable than a longer waiting time (would sometimes return empty values for bid/ask info). Like you said, there could be at most ~6 updates/second. Can we achieve that using functions like "testMkDataCallbacks2()", or it should be some other ways?

Thanks! -Alex

function dat = testMkDataCallbacks2()

% get TWS session instance session = TWS.Session.getInstance();

% create a callback to print error messages to the command window lherr = event.listener( ... TWS.Events.getInstance ,... TWS.Events.ERROR ,... @(s,e)disp(e.event.data) ... ); clientId = 2;

% connect to TWS session.eClientSocket.eConnect('127.0.0.1',7496,clientId);

contract = com.ib.client.Contract(); contract.m_symbol = 'SPY' ; contract.m_exchange = 'ARCA'; contract.m_primaryExch = 'ARCA'; contract.m_secType = 'STK' ; contract.m_currency = 'USD' ;

% list of metadata ticks genericTickList = ['225']; % get realtime auction data

dat = struct(); dat.bidsize = []; dat.bidprice = []; dat.asksize = []; dat.askprice = []; dat.lastsaleprice = []; dat.lastsalesize = []; dat.totalvolume = [];

function mkDatCallback(e)

switch e.data.tickId

    case 0
        % bid size
        dat.bidsize(end+1) = e.data.value;
    case 1
        % bid price
        dat.bidprice(end+1) = e.data.value;
    case 2
        % ask price
        dat.askprice(end+1) = e.data.value;
    case 3
        % ask size
        dat.asksize(end+1) = e.data.value;
    case 4
        % last sale price
        dat.lastsaleprice(end+1) = e.data.value;
    case 5
        % last sale size
        dat.lastsalesize(end+1) = e.data.value;
    case 8
        % total volume
        dat.totalvolume(end+1) = e.data.value;
    otherwise
        % no op
        disp(['no switch case for this tickId: ' num2str(e.data.tickId)])
end

end

% create a callback to update gui datlh = event.listener( ... TWS.Events.getInstance ,... TWS.Events.MARKETDATA ,... @(s,e)mkDatCallback(e.event) ... );

% create unique id for this mk data request reqId = 0;

% request market data from TWS session.eClientSocket.reqMktData(reqId,contract,genericTickList,false,[]);

% wait while data is collected pause(0.2);

% now that we have data, tell TWS we don't need more data session.eClientSocket.cancelMktData(reqId);

delete(lherr); delete(datlh); end

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/8#issuecomment-309786813, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1J3Y4hUo4OibZ-erSwdDTag8sqPrks5sF97dgaJpZM4NtqM3.