softwarespartan
commented
7 years ago Thanks for reaching out. Apologies, for delay in getting back to you.
First you can use the script below for grabbing historical data. I need to add it to the repository. Here is the link https://www.dropbox.com/s/qdxzhyneg734auz/historicDataQuery.m?dl=0.
You can also subscribe to “market bar data” which is every 5 seconds.
I just create a listener that consolidates the 5 second market bar data and then publishes a new bar to all the listeners at whatever desired frequency.
BTW, you code looks great — well done!
Regarding accessing the fields of an object, you can always ask for “fields” and “methods”
For example:
When you create an order object in matlab.
order = com.tws.OrderFactory.GenericLimitOrder('DU207406', 'BUY', 100, 198.5);
See the IB docs on com.ib.client.Order https://www.interactivebrokers.com/en/software/api/apiguide/java/order.htm for more info on each order property.
Then use the matlab function “fieldnames()” to get a list of order properties.
fieldnames(order)
ans =
'CUSTOMER'
'FIRM'
'OPT_UNKNOWN'
'OPT_BROKER_DEALER'
'OPT_CUSTOMER'
'OPT_FIRM'
'OPT_ISEMM'
'OPT_FARMM'
'OPT_SPECIALIST'
'AUCTION_MATCH'
'AUCTION_IMPROVEMENT'
'AUCTION_TRANSPARENT'
'EMPTY_STR'
'm_orderId'
'm_clientId'
'm_permId'
'm_action'
'm_totalQuantity'
'm_orderType'
'm_lmtPrice'
'm_auxPrice'
'm_tif'
'm_activeStartTime'
'm_activeStopTime'
'm_ocaGroup'
'm_ocaType'
'm_orderRef'
'm_transmit'
'm_parentId'
'm_blockOrder'
'm_sweepToFill'
'm_displaySize'
'm_triggerMethod'
'm_outsideRth'
'm_hidden'
'm_goodAfterTime'
'm_goodTillDate'
'm_overridePercentageConstraints'
'm_rule80A'
'm_allOrNone'
'm_minQty'
'm_percentOffset'
'm_trailStopPrice'
'm_trailingPercent'
'm_faGroup'
'm_faProfile'
'm_faMethod'
'm_faPercentage'
'm_openClose'
'm_origin'
'm_shortSaleSlot'
'm_designatedLocation'
'm_exemptCode'
'm_discretionaryAmt'
'm_eTradeOnly'
'm_firmQuoteOnly'
'm_nbboPriceCap'
'm_optOutSmartRouting'
'm_auctionStrategy'
'm_startingPrice'
'm_stockRefPrice'
'm_delta'
'm_stockRangeLower'
'm_stockRangeUpper'
'm_volatility'
'm_volatilityType'
'm_continuousUpdate'
'm_referencePriceType'
'm_deltaNeutralOrderType'
'm_deltaNeutralAuxPrice'
'm_deltaNeutralConId'
'm_deltaNeutralSettlingFirm'
'm_deltaNeutralClearingAccount'
'm_deltaNeutralClearingIntent'
'm_deltaNeutralOpenClose'
'm_deltaNeutralShortSale'
'm_deltaNeutralShortSaleSlot'
'm_deltaNeutralDesignatedLocation'
'm_basisPoints'
'm_basisPointsType'
'm_scaleInitLevelSize'
'm_scaleSubsLevelSize'
'm_scalePriceIncrement'
'm_scalePriceAdjustValue'
'm_scalePriceAdjustInterval'
'm_scaleProfitOffset'
'm_scaleAutoReset'
'm_scaleInitPosition'
'm_scaleInitFillQty'
'm_scaleRandomPercent'
'm_scaleTable'
'm_hedgeType'
'm_hedgeParam'
'm_account'
'm_settlingFirm'
'm_clearingAccount'
'm_clearingIntent'
'm_algoStrategy'
'm_algoParams'
'm_whatIf'
'm_notHeld'
'm_smartComboRoutingParams'
'm_orderComboLegs'
'm_orderMiscOptions'Don’t hesitate to ask if you need anything additional. Let me know if you have any trouble accessing variables etc. I can send more examples.
Cheers -abel
function [s,bars] = (contract,dt,config)
% some randome number for unique request id r = round(rand*10000);
% get TWS session session = TWS.Session.getInstance();
% connect the session to TWS running instance session.eClientSocket.eConnect('127.0.0.1',7496,0);
% create local buffer for historical data events [buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA);
% loop over number of requests for i = 1:config.N
% create unique id
requestId = r+100000+i;
% compute the end time of the request
%endDateTime = datestr(dt-config.delta*(i-1),'yyyymmdd HH:MM:SS');
endDateTime = datestr(dt-config.delta*(i-1),'yyyymmdd 16:00:00');
% make the request for data
session.eClientSocket.reqHistoricalData( ...
requestId ,...
contract ,...
endDateTime ,...
config.duration ,...
config.barsize ,...
config.whatToShow ,...
config.useRTH ,...
1 ,...
[] ...
);
pause(0.5);end
% wait around for the data pause(10);
% init hde hde = cell(1,buf.size);
% Retreive the historical data event from the buffer for i = 1:buf.size; hde{i} = buf.get(); buf.remove(); end
% Convert the HashSet
% Extract close and high low ave info from each bar`
close = cellfun(@(b)b.close ,bars);
vol = cellfun(@(b)b.volume,bars);
high = cellfun(@(b)b.high ,bars);
low = cellfun(@(b)b.low ,bars);
open = cellfun(@(b)b.open ,bars);
wap = cellfun(@(b)b.wap ,bars);
count = cellfun(@(b)b.count ,bars);
hla = cellfun(@(b)(b.high+b.low)/2,bars);
% Convert the string time of the bars to matlab datenum dt = datenum(cellfun(@(b)char(b.dtstr),bars,'UniformOutput',false),'yyyymmdd HH:MM:SS');
% The bars might not be in order once extracted from the HashSet [~,indx] = sort(dt);
% sort time stamps and get unique dt = dt(indx); [~,uix] = unique(dt);
% sort and assign
s.dt = dt (uix);
close = close(indx); s.close = close(uix);
hla = hla (indx); s.hla = hla (uix);
high = high (indx); s.high = high (uix);
low = low (indx); s.low = low (uix);
open = open (indx); s.open = open (uix);
vol = vol (indx); s.volume = vol (uix);
wap = wap (indx); s.wap = wap (uix);
count = count(indx); s.count = count(uix);
% sort the bars bars = bars(indx); bars = bars(uix);
% clean up listener handle cellfun(@(l)delete(l),lh);
On Jul 8, 2017, at 5:10 PM, ivs17 notifications@github.com wrote:
Hey,
thx Abel for this tool and redirecting
my Q here,
I tested the instructions to get contract details on this page: http://softwarespartan.github.io/IB4m/docs/html/ContractDetailsExample.html http://softwarespartan.github.io/IB4m/docs/html/ContractDetailsExample.html As shown on that page the contract details do get printed on the command line. Now - My problem is that I can not get my hands specific variables.
But this is not an issue but maybe rather my limited knowledge on classes in java.
For example I wrote a script to download 1 min data and combine them. Since IB also offers historical data for already expired futures contracts it would be handy to limit the dates for a historical data request to the last trading date.
For now I found a work around with a 5x1 "moving" boleans which break the request-for loop. 5 x no data in a row --> move to next contract.
Since some requests result in an error anyway I have three more script to
identify missing data try a second download for them 3 weld the data with previously downloaded data Will post them as well Attached: m-File with Code ( better formatting than here) upload.zip https://github.com/softwarespartan/IB4m/files/1133317/upload.zip Here is the code I use. Pls don't pick on me on variable names xD %================================================
%% Building an 1 min database for an underlying % credit to Abel Braun - publisher of IB4m
% add TWS.jar cd('E:\MatlabFunktionen\softwarespartan-IB4m-aff30d8'); javaaddpath(fullfile(pwd,'Jar','TWS.jar'));
%% initialize session with TWS session = TWS.Session.getInstance();
%% connect to TWS session.eClientSocket.eConnect('127.0.0.1',7496,0); pause(0.5);
%% Create datevector for downloads
% no Quote booster var 365d 1y % See " Download Limitation in IB API reference for details var=365; dtFutures=busdays(round(now()-var),round(now())); dtFutures=datestr(dtFutures,'yyyymmdd');
% IB seems to use usertime instead of exchange time % 23:00:00 is the closing time for Central Europe % dtFutures=[dtFutures,repmat(' 23:00:00',numel(dtFutures(:,1)),1)];
% create CME codes in this case for crude oil mainunderlying="CL"; mc={'F','G','H','J','K','M','N','Q','U','V','X','Z'}; %CME Month Codes m={'01','02','03','04','05','06','07','08','09','10','11','12'}; % Months
%this one is tricky since and depends on the amount of % quote boosters / comissions % and the underlying ( for Crude Oil currently (08.07.2017) CLZ"2025" startD=2016; endD=2025;
delta=(endD-startD); Esymbols=cell(delta,1);
for i= 0:delta-1 for y=1:12 Esymbols{y+i*12}=[num2str(startD+i),m{y}]; end end clear startD endD delta i y %% % delete undownloadadable futures (expired 1y ago and earlier) % depends on your quote boosters / comissions
oneyago=round(now())-365+30; oneyagoS=datenum(Esymbols,'yyyymm'); helper=oneyagoS(oneyagoS>oneyago,1); helper=datestr(helper,'yyyymm'); Esymbols=cellstr(helper);
clear helper oneyago onyagoS
%% Mainloop for omikron=1:numel(Esymbols)
% direcotry where IB4M was added % might be unessacary cd('E:\MatlabFunktionen\softwarespartan-IB4m-aff30d8');
%my standard name for structs savezone=struct(); disp([num2str(omikron),' / ', num2str(numel(Esymbols)),... ' ',Esymbols{omikron},' ' datestr(now(),'HH:MM:SS')]);
%% create an contract and initialize it for crude oil for contractwrapper=1
contract = com.ib.client.Contract(); contract.m_symbol = mainunderlying ; % String oder Char? contract.m_secType = 'FUT' ; contract.m_exchange = 'NYMEX'; contract.m_currency = 'USD' ; contract.m_expiry=Esymbols{omikron}; contract.m_includeExpired =true;
% pacing info
% Duration Allowed Bar Sizes % 60 S 1 secs - 1 mins % 120 S 1 secs - 2 mins % 1800 S (30 mins) 1 secs - 30 mins % 3600 S (1 hr) 5 secs - 1 hr % 14400 S (4hr) 10 secs - 3 hrs % 28800 S (8 hrs) 30 secs - 8 hrs % 1 D 1 min - 1 day % 2 D 2 mins - 1 day % 1 W 3 mins - 1 week % 1 M 30 mins - 1 month % 1 Y 1 day - 1 month
% posobility to ask for more datatyps, see table below % simplification: only trades datatypeS={"TRADES",... "MIDPONT",... "BID",... "ASK",... "BID_ASK",... "HISTORICAL_VOLATILITY",... "OPTION_IMPLIED_VOLATILITY"};
datatype="TRADES";
% MIDPOINT BID_ASK % Product TRADES | BID ASK | HV IV % Stocks Y Y Y Y Y Y Y % Commodities N Y Y Y Y N N % Options Y Y Y Y Y N N % Futures Y Y Y Y Y N N % FOPs Y Y Y Y Y N N % ETFs Y Y Y Y Y Y Y % Warrants Y Y Y Y Y N N % Structured Y Y Y Y Y N N % SSFs Y Y Y Y Y N N % Forex N Y Y Y Y N N % Metals Y Y Y Y Y N N % Indices Y N N N N Y Y % Bonds Y Y Y Y Y N N % Funds N Y Y Y Y N N % CFDs* N Y Y Y Y N N end %contractwrapper % Get contract Details %
% % Create Buffer for contract Details % [buf2,lh2] = TWS.initBufferForEvent(TWS.Events.CONTRACTDETAILS); % % % request contract details % session.eClientSocket.reqContractDetails(0,contract); % pause(1); % % % % copy contract details % cdtls= buf2.get().data; % pause(0.5); % WTF NOW ? -.-'
% error count - exit if 5x false erc=true(5,1);
for vega=1 %wrapper or var for different data types for i=1:numel(dtFutures(:,1))
% error count - exit if 5x false if sum(erc)==0 break end
try %% create local buffer for historical data events [buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA); pause(1) %% Request data disp([Esymbols{omikron},' ',dtFutures(i,1:end)]); % contract % time for last candle % how much data (see pacing info above) % granularity ( see pacing info above= % datatype TRADES BID ASK ect.. % 0 include date out of RTH | 1 only RTH % don't know % don't know
session.eClientSocket.reqHistoricalData(i,... contract,... dtFutures(i,1:end),... '1 D',... '1 min',... datatype,... 0,... 1,... []);%% Retreive the historical data event from the buffer
% depending on your connection to IB servers and machine % I set the pause for 3 seconds to reduce errors pause(3); try hde = buf.get(); catch % try a second time and disp the error to command line disp('Buffer error'); pause(3) hde = buf.get(); end %% Convert the HashSet into more native cell array of bars
bars = collection2cell(hde.data);
%% Extract close and volume info from each bar
open= cellfun(@(b)b.open,bars); high = cellfun(@(b)b.high,bars); low = cellfun(@(b)b.low,bars); close = cellfun(@(b)b.close,bars); volume = cellfun(@(b)b.volume,bars);
%% Convert the string time of the bars to matlab datenum dt = datenum(cellfun(@(b)char(b.dtstr),bars,'UniformOutput',false),'yyyymmdd HH:MM:SS');
% The bars might not be in order once extracted from the HashSet [~,indx] = sort(dt);
% Concatenate exisisting and new data for contwrapper=1
if i==1 %wirst request savezone.prices.dt=dt(indx); savezone.prices.open=open(indx); savezone.prices.high=high(indx); savezone.prices.low=low(indx); savezone.prices.close=close(indx); savezone.prices.volume=volume(indx); else % next requests % check if prices struct exists % in case first request resultet in an error fn= fieldnames(savezone); pcheck=ismember('prices',fn); if pcheck== true % Data DOES exist savezone.prices.dt=[savezone.prices.dt;dt(indx)]; savezone.prices.open=[savezone.prices.open;open(indx)]; savezone.prices.high=[savezone.prices.high;high(indx)]; savezone.prices.low=[savezone.prices.low;low(indx)]; savezone.prices.close=[savezone.prices.close;close(indx)]; savezone.prices.volume=[savezone.prices.volume;volume(indx)]; else % Data does NOT exist savezone.prices.dt=dt(indx); savezone.prices.open=open(indx); savezone.prices.high=high(indx); savezone.prices.low=low(indx); savezone.prices.close=close(indx); savezone.prices.volume=volume(indx); end end% moving Error counter % copy results frin last 4 attempts % and write a true if succeeded.....or erc(1:4,1)=erc(2:5,1); erc(5,1)=true; end % contwrapper catch
% and write a false if failed disp('error trying to download day'); erc(1:4,1)=erc(2:5,1); erc(5,1)=false;end clear buf lh hde bars open high low close volume ans
end % for i daycount end % vega
%% Saving downloaded data
for savewrapper=1 % cd to folder with data % change if you want cd('E:\Markets\prices\Futures'); a=exist(char(mainunderlying),'dir');
% Check if subfolder for Underlying existsif a~=7 mkdir(char(mainunderlying)); end % Switch to directory of underlying and save try cd(char(mainunderlying)); % Underlying | Contractmonth and Year | free... save([char(mainunderlying),Esymbols{omikron},'.Prices.minute.mat'],'savezone'); catch disp("folder Error"); pause(2); % second attempt mkdir(char(mainunderlying)); cd(char(mainunderlying)); save([char(mainunderlying),Esymbols{omikron},'.Prices.minute.mat'],'savezone'); end end %savewrapper disp('error trying to download day');
end % omikron clear
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/softwarespartan/IB4m/issues/10, or mute the thread https://github.com/notifications/unsubscribe-auth/AEWq1NxmKcXIiHUHtRc-mPNbcByGsneFks5sL_BfgaJpZM4OR3nm.
ghost
NewbieAlex
%EDIT of course there are other workarounds like downloading a year of daily data to get the exact dates. upload2.zip %EDIT Hey,
thx Abel for this tool and redirecting
my Q here,
I tested the instructions to get contract details on this page: http://softwarespartan.github.io/IB4m/docs/html/ContractDetailsExample.html
As shown on that page the contract details do get printed on the command line. Now - My problem is that I can not get my hands specific variables.
But this is not an issue but maybe rather my limited knowledge on classes in java.
For example I wrote a script to download 1 min data and combine them. Since IB also offers historical data for already expired futures contracts it would be handy to limit the dates for a historical data request to the last trading date.
For now I found a work around with a 5x1 "moving" boleans which break the request-for loop. 5 x no data in a row --> move to next contract.
Since some requests result in an error anyway I have three more script to
Attached: m-File with Code ( better formatting than here) upload.zip
Here is the code I use. Pls don't pick on me on variable names xD %================================================
%% Building an 1 min database for an underlying % credit to Abel Braun - publisher of IB4m
% add TWS.jar cd('E:\MatlabFunktionen\softwarespartan-IB4m-aff30d8'); javaaddpath(fullfile(pwd,'Jar','TWS.jar'));
%% initialize session with TWS session = TWS.Session.getInstance();
%% connect to TWS session.eClientSocket.eConnect('127.0.0.1',7496,0); pause(0.5);
%% Create datevector for downloads
% no Quote booster var 365d 1y % See " Download Limitation in IB API reference for details var=365; dtFutures=busdays(round(now()-var),round(now())); dtFutures=datestr(dtFutures,'yyyymmdd');
% IB seems to use usertime instead of exchange time % 23:00:00 is the closing time for Central Europe % dtFutures=[dtFutures,repmat(' 23:00:00',numel(dtFutures(:,1)),1)];
% create CME codes in this case for crude oil mainunderlying="CL"; mc={'F','G','H','J','K','M','N','Q','U','V','X','Z'}; %CME Month Codes m={'01','02','03','04','05','06','07','08','09','10','11','12'}; % Months
%this one is tricky since and depends on the amount of % quote boosters / comissions % and the underlying ( for Crude Oil currently (08.07.2017) CLZ"2025" startD=2016; endD=2025;
delta=(endD-startD); Esymbols=cell(delta,1);
for i= 0:delta-1 for y=1:12 Esymbols{y+i*12}=[num2str(startD+i),m{y}]; end end clear startD endD delta i y %% % delete undownloadadable futures (expired 1y ago and earlier) % depends on your quote boosters / comissions
oneyago=round(now())-365+30; oneyagoS=datenum(Esymbols,'yyyymm'); helper=oneyagoS(oneyagoS>oneyago,1); helper=datestr(helper,'yyyymm'); Esymbols=cellstr(helper);
clear helper oneyago onyagoS
%% Mainloop for omikron=1:numel(Esymbols)
for contractwrapper=1
contract = com.ib.client.Contract(); contract.m_symbol = mainunderlying ; % String oder Char? contract.m_secType = 'FUT' ; contract.m_exchange = 'NYMEX'; contract.m_currency = 'USD' ; contract.m_expiry=Esymbols{omikron};
contract.m_includeExpired =true;
% pacing info
% Duration Allowed Bar Sizes % 60 S 1 secs - 1 mins % 120 S 1 secs - 2 mins % 1800 S (30 mins) 1 secs - 30 mins % 3600 S (1 hr) 5 secs - 1 hr % 14400 S (4hr) 10 secs - 3 hrs % 28800 S (8 hrs) 30 secs - 8 hrs % 1 D 1 min - 1 day % 2 D 2 mins - 1 day % 1 W 3 mins - 1 week % 1 M 30 mins - 1 month % 1 Y 1 day - 1 month
% posobility to ask for more datatyps, see table below % simplification: only trades datatypeS={"TRADES",... "MIDPONT",... "BID",... "ASK",... "BID_ASK",... "HISTORICAL_VOLATILITY",... "OPTION_IMPLIED_VOLATILITY"};
datatype="TRADES";
% MIDPOINT BID_ASK % Product TRADES | BID ASK | HV IV % Stocks Y Y Y Y Y Y Y % Commodities N Y Y Y Y N N % Options Y Y Y Y Y N N % Futures Y Y Y Y Y N N % FOPs Y Y Y Y Y N N % ETFs Y Y Y Y Y Y Y % Warrants Y Y Y Y Y N N % Structured Y Y Y Y Y N N % SSFs Y Y Y Y Y N N % Forex N Y Y Y Y N N % Metals Y Y Y Y Y N N % Indices Y N N N N Y Y % Bonds Y Y Y Y Y N N % Funds N Y Y Y Y N N % CFDs* N Y Y Y Y N N end %contractwrapper % Get contract Details %
% % Create Buffer for contract Details % [buf2,lh2] = TWS.initBufferForEvent(TWS.Events.CONTRACTDETAILS); %
% % request contract details % session.eClientSocket.reqContractDetails(0,contract); % pause(1); %
%
% % copy contract details % cdtls= buf2.get().data;
% pause(0.5); % WTF NOW ? -.-'
% error count - exit if 5x false erc=true(5,1);
for vega=1 %wrapper or var for different data types for i=1:numel(dtFutures(:,1))
%% create local buffer for historical data events [buf,lh] = TWS.initBufferForEvent(TWS.Events.HISTORICALDATA); pause(1) %% Request data disp([Esymbols{omikron},' ',dtFutures(i,1:end)]); % contract % time for last candle % how much data (see pacing info above) % granularity ( see pacing info above= % datatype TRADES BID ASK ect.. % 0 include date out of RTH | 1 only RTH % don't know % don't know
%% Retreive the historical data event from the buffer
% depending on your connection to IB servers and machine % I set the pause for 3 seconds to reduce errors pause(3); try hde = buf.get(); catch % try a second time and disp the error to command line disp('Buffer error'); pause(3) hde = buf.get(); end %% Convert the HashSet into more native cell array of bars
bars = collection2cell(hde.data);
%% Extract close and volume info from each bar
open= cellfun(@(b)b.open,bars); high = cellfun(@(b)b.high,bars); low = cellfun(@(b)b.low,bars); close = cellfun(@(b)b.close,bars); volume = cellfun(@(b)b.volume,bars);
%% Convert the string time of the bars to matlab datenum dt = datenum(cellfun(@(b)char(b.dtstr),bars,'UniformOutput',false),'yyyymmdd HH:MM:SS');
% The bars might not be in order once extracted from the HashSet [~,indx] = sort(dt);
% Concatenate exisisting and new data for contwrapper=1
end % contwrapper catch
clear buf lh hde bars open high low close volume ans
end % for i daycount end % vega
%% Saving downloaded data
for savewrapper=1 % cd to folder with data % change if you want cd('E:\Markets\prices\Futures\'); a=exist(char(mainunderlying),'dir');
% Switch to directory of underlying and save try cd(char(mainunderlying)); % Underlying | Contractmonth and Year | free... save([char(mainunderlying),Esymbols{omikron},'.Prices.minute.mat'],'savezone'); catch disp("folder Error"); pause(2); % second attempt mkdir(char(mainunderlying)); cd(char(mainunderlying)); save([char(mainunderlying),Esymbols{omikron},'.Prices.minute.mat'],'savezone'); end end %savewrapper disp('error trying to download day');
end % omikron clear