IQFeed project questions

[alexveden]

@spickering-git I'm looking how to integrate IQFeed into the current datafeed structure. Do you have any thoughts how to smoothly replace CQG in current setup? I can try just fill 'futurebarcol' collection in the 'tmldb_test' DB, but probably you might have another point of view?

Several questions to discuss

1. Are there any other collections which required to be filled by IQFeed?

2. How can you envisage the ticker name mapping i.e. CQG -> IQFeed. The problem that IQFeed has own ticker root notation. For example, Crude futures have a QCL*** root. I may add new IQFeed root field to 'instruments' collection to address this. @spickering-git , what is your opinion?

3. Can you confirm that all dates in 'futurebarcol' are in PST? IQFeed date|times are in EST, and I need to implement explicit conversion.

4. Should I implement holiday calendar filter for IQFeed? Or you handle this somewhere else?

5. As I remember IQFeed 1-min intervals had a weird time shift. For example, bars which were built using ticks in the date range of 10:00:00.0000 - 10:00:59.9999 appeared in the history with timestamp 10:01:00. I must fix it too, but before this, I would like to know how you manage timestamps in CQG datafeed?

6. What is the role of 'contractcol' collection in the 'tmldb_test' DB? Especially I'm wondering about 'cqgsymbol' field.

[spickering-git]

1

Are there any other collections which required to be filled by IQFeed?

a. CQG fills futurebarcol in tmldb_test live through the day for the model, it only fills a few of the futures contracts for each instrument (I will need to check but I think the front month out to the next 4 contracts) .

b. The realtime connects directly to the CQG api and makes calls for future and option data. We would need to rework the realtime to point to a live data collection that is being filled with IQFeed data. We could point the realtime to futurebarcol, but would need to decide if we want to get the live option data. The realtime uses a combination of calculating option prices and subscribing to live option bid and asks. Often the bid and asks are unreasonable so a price is calculated from yesterdays implied vol.

c. CQG fills the contracts_bar in tmldb_v2 for all of the available futures at ~ 13:00 PST every day.

The futures data in futurebarcol in tmldb is filled into the tmqr2 quotes_intraday collection.

Possibly you have an idea for integrating data coming in from iqfeed for both V1 and V2. Or we can just keep it the way it is now and essentially just use iqfeed to fill data the way CQG is now.

2

How can you envisage the ticker name mapping i.e. CQG -> IQFeed. The problem that IQFeed has own ticker root notation.

The symbology in iqfeed looks pretty straight forward. CQG has its own ticker root notation also. The span collection software generates the CQG symbol so that the call to CQG is easy. Can't we just do this on the fly?

3

Can you confirm that all dates in 'futurebarcol' are in PST? IQFeed date|times are in EST, and I need to implement explicit conversion.

Yes, all dates times are in PST in futurebarcol

4

Should I implement holiday calendar filter for IQFeed? Or you handle this somewhere else?

The code that fills futurebarcol from CQG doesn't have any filter for holidays.

When the tmqr2 quotes_intraday is filled, this script uses the holiday calendar to filter data

5

As I remember IQFeed 1-min intervals had a weird time shift. For example, bars which were built using ticks in the date range of 10:00:00.0000 - 10:00:59.9999 appeared in the history with timestamp 10:01:00. I must fix it too, but before this, I would like to know how you manage timestamps in CQG datafeed?

CQG timestamps are in PST

6

What is the role of 'contractcol' collection in the 'tmldb_test' DB? Especially I'm wondering about 'cqgsymbol' field.

contractcol is updated with the futures contracts that we ask for data for from CQG. The cqgsymbol is taken from the 'contracts' collection in 'tmldb_v2' which is filled by combining the instrument label, the month code and the year.

[alexveden]

I've added instrument map for framework instrument to IQFeed. Could you recheck the validity of exchanges? Because the same products trade on different exchanges and it's difficult to understand for me which is correct.

https://10.0.1.2:8889/notebooks/data/IQFeed%20Instrument%20Map.ipynb

[nikolas-joyce]

Where Can I find the map. Thx

Sent from my iPhone

On Dec 8, 2017, at 9:06 PM, alexveden notifications@github.com<mailto:notifications@github.com> wrote:

I've added instrument map for framework instrument to IQFeed. Could you recheck the validity of exchanges? Because the same products trade on different exchanges and it's difficult to understand for me which is correct.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftrendmanagement%2FTmqr-framework-2%2Fissues%2F96%23issuecomment-350424514&data=02%7C01%7C%7C8ffc3aa689304394c4d408d53ec29491%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636483927785643199&sdata=TvoJToqaEJTnb62cvFyclbYz30vSZMq5w6gN6tBMQro%3D&reserved=0, or mute the threadhttps://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARobLAoehW94kSxL2aGx08xU-QjltBUCks5s-hVIgaJpZM4Q5Abe&data=02%7C01%7C%7C8ffc3aa689304394c4d408d53ec29491%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636483927785643199&sdata=mw0PoOYNxDCjCMKhSaTpfMnHUII48UN10J0rKcYQWiI%3D&reserved=0.

[alexveden]

sorry, I've edited the post and added the link

Отправлено из Mail.Ru для Android суббота, 09 декабря 2017г., 11:09 +04:00 от NikolasJoyce notifications@github.com :

Where Can I find the map. Thx

Sent from my iPhone

On Dec 8, 2017, at 9:06 PM, alexveden <notifications@github.com<mailto: notifications@github.com >> wrote:

I've added instrument map for framework instrument to IQFeed. Could you recheck the validity of exchanges? Because the same products trade on different exchanges and it's difficult to understand for me which is correct.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub< https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftrendmanagement%2FTmqr-framework-2%2Fissues%2F96%23issuecomment-350424514&data=02%7C01%7C%7C8ffc3aa689304394c4d408d53ec29491%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636483927785643199&sdata=TvoJToqaEJTnb62cvFyclbYz30vSZMq5w6gN6tBMQro%3D&reserved=0 >, or mute the thread< https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARobLAoehW94kSxL2aGx08xU-QjltBUCks5s-hVIgaJpZM4Q5Abe&data=02%7C01%7C%7C8ffc3aa689304394c4d408d53ec29491%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636483927785643199&sdata=mw0PoOYNxDCjCMKhSaTpfMnHUII48UN10J0rKcYQWiI%3D&reserved=0 >. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub , or mute the thread .

[alexveden]

I'm nearby the finish line of IQFeed data module, here is a brief algorithm how it works:

1. Get list of futures to update (front month + 3 contracts) for all instruments in the DB
2. Get last quote date in the v2 DB for each contract and request backfill
3. Update live bars in v2 and v1 (futurebarcol in live v1 Mongo DB) simultaneously

Couple questions about deployment for @spickering-git:

1. Can you confirm that contractid field of 'futurebarcol' in 'tmldb_test' equal and aligned to original ID in 'tmldb_v2' database?
2. I can deploy the IQFeed directly to Linux VM machine. IQFeed has worked pretty stable on my local Linux machine and we have an option for deployment without utilizing windows VM. Do you have any concerns/objections about this?

Before deploying it into production I'm going to create a couple separate collections for v1 and v2 quotes data formats and create a notebook for quotes comparison. Once the quotes to be considered as aligned we can run this stuff in production.

[spickering-git]

updated 'SB':'@ SB' #SUGAR #11 WORLD MARCH 2018 ICEFU FUTURE ICEFU 'CC':'@ CC' #COCOA DECEMBER 2017 ICEFU FUTURE ICEFU

[spickering-git]

Can you confirm that contractid field of 'futurebarcol' in 'tmldb_test' equal and aligned to original ID in 'tmldb_v2' database?

Yes the idcontract field in futurebarcol is the same as the idcontract in tmldb_v2 database.

I can deploy the IQFeed directly to Linux VM machine. IQFeed has worked pretty stable on my local Linux machine and we have an option for deployment without utilizing windows VM. Do you have any concerns/objections about this?

That sounds like it would be the most efficient.

However, if you are filling 'futurebarcol' in 'tmldb_test', live through the day, what do you think about filling all of the contracts 'contract_bars' in 'tmldb_v2 ' at the end of the day? Do you think that it worth while?

Since this will now be our primary datasource, I will need to change the realtime to point to this futures data. Do you have an idea of how we could get the options data from iqfeed into the realtime? Before we actually get any option data going I will just calculate the option prices from yesterdays settlement implied vol.

[alexveden]

However, if you are filling 'futurebarcol' in 'tmldb_test', live through the day, what do you think about filling all of the contracts 'contract_bars' in 'tmldb_v2 ' at the end of the day? Do you think that it worth while?

What about getting rid of 'futurebarcol' in the live DB and use 'contract_bars' in 'tmldb_v2 ' for live calculations?

Do you have an idea of how we could get the options data from iqfeed into the realtime? Before we actually get any option data going I will just calculate the option prices from yesterdays settlement implied vol.

I can create the online collection of bids/asks/last for options and futures. But the problem is that IQFeed has online symbol limit about 500 tickers, almost 80 symbols are used for futures. So I don't think that we can fit this limit. However, we can extend this limit up to 2000 (as I remember correctly) for an extra fee.

[alexveden]

Alternatively, you can connect to IQFeed client (via local network) and request required options data updates.

[spickering-git]

To make note of this

Get list of futures to update (front month + 3 contracts) for all instruments in the DB

I think this should be specific for each instrument currencies for example have futures for every month now but we are only trading quarterlies if we are collecting 6E for example the contracts would include Dec 17, Jan 18, Feb 18, Mar 18 but if Nik does an offset where he wants to trade another contract out we would need the Jun18 So i think we should ask for front month + 6

But the # of contracts collected should be specified for each instrument

[spickering-git]

What about getting rid of 'futurebarcol' in the live DB and use 'contract_bars' in 'tmldb_v2 ' for live calculations?

My initial reaction was not right away, but after further thought this makes more sense. We are simply filling the contract_bars through the day and eliminating the end of day run.

The only issue is if there is a mis-priced bar, that will stay in our historical data collection.

I think we should get rid of 'futurebarcol' in the live DB and use 'contract_bars' in 'tmldb_v2 ' for live calculations.

@alexveden do you agree with this approach?

[alexveden]

@alexveden do you agree with this approach?

totally

Brief TODO for IQFeed project for next week:

1. Add holiday filter to v2 collection writing
2. Investigate possible bug for ES/CL backfill/initiation
3. Add v1 collection matching for https://10.0.1.2:8889/notebooks/data/IQFeed%20data%20monitoring.ipynb
4. Make deployment scripts (cronned restart, supervisor settings)
5. Rewrite the v1 format to match 'contract_bars' in 'tmldb_v2 ' for live

[spickering-git]

@alexveden Can we have iqfeed pull in just the front month of the eurodollar futures and fill the database with the settlement or close for each single day? Then we will have the updated risk free rate. I'm not sure right now if you use the legacy collection in tmldb_v2 'option_input_data'. If not we can just fill the quotes_riskfreerate collection. We need the latest risk free rate for the realtime.

[alexveden]

I have a feeling that IQFeed prevents too frequent historical requests, for same tickers. I.e. struggles with delayed (live) updates/polls. When I restart the script, it requests IQFeed but no answer sent back from the IQFeed client. I've checked this on the low level, to exclude bugs in my code.

[spickering-git]

@alexveden Tomorrow we will have to speak to iqfeed support. On a windows install of iqfeed there is a diagnostics installed that they asked us to do. I think for testing and diagnostic purposes we should possibly run the iqfeed data collection on the Windows VM to enable their diagnostics.

[alexveden]

After double checking the problem, I think that it's a Linux emulator problem. Historical requests sometimes crashes iqconnect.exe and it hangs. But the sockets remain active, this lead to zombie behavior which is very hard to catch.

I will keep trying to make it work on Linux, but we must have backup plan to launch it on windows VM.

I've added 'killall iqconnect.exe' to supervisor script, it might help.

[alexveden]

Also, this behavior occurs only at the initialization phase, once the iqfeed is initiated it works fine. Historical-live updates for 3 products take about 3.5 seconds, so I think it might be acceptable for live updates. The refresh interval is 2 minutes, and now it's updating 3 products: ES, CL, NG.

[alexveden]

@spickering-git is there some information that I need to know regarding the conversion of Eurodollar future prices to risk free rate values?

[nikolas-joyce]

You subtract the Eurodollar from 100 to get the risk free rate. If Eurodollar is 98 then 100-98 means our risk free rate proxy is 2%.

Sent from my iPhone

On Dec 26, 2017, at 7:46 PM, alexveden notifications@github.com<mailto:notifications@github.com> wrote:

@spickering-githttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fspickering-git&data=02%7C01%7C%7C16e76749352a4b07145508d54cdc74dc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636499432088995200&sdata=zdd%2BZJUVJJ7AmVgn%2FDoAteaGtvz2S6ha341c%2B6pgKcY%3D&reserved=0 is there some information that I need to know regarding the conversion of Eurodollar future prices to risk free rate values?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHubhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftrendmanagement%2FTmqr-framework-2%2Fissues%2F96%23issuecomment-354047002&data=02%7C01%7C%7C16e76749352a4b07145508d54cdc74dc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636499432089151441&sdata=nhhBVflcs1bSVSZoj0NgNAjq9CqINLOBDGPCGIwq%2Fiw%3D&reserved=0, or mute the threadhttps://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARobLI7bkk4ZsDG_yFhMq_9guwXNcZbdks5tEb2lgaJpZM4Q5Abe&data=02%7C01%7C%7C16e76749352a4b07145508d54cdc74dc%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636499432089151441&sdata=Q8ssNknFBKcNO2IYr3izkrS6whQJYZMb0rtrgXgEQxs%3D&reserved=0.

[alexveden]

@spickering-git I saw that you have added new tickers to the IQFeed, but some of them are not available for updates because the asset index doesn't contain information about futures series.

You should populate asset index by futures information to make live updates work.

Also, take a look: https://10.0.1.2:8889/notebooks/data/IQFeed%20Instrument%20Map.ipynb

I've added new settings for each product, which represents update mode (historical polls or real-time):

[alexveden]

@spickering-git regarding 'futurebarcol' -> 'contract_bars' collection switch:

1. I'm ready to do this on your mark
2. Can you confirm that contract_bars collection is up to date and aligned with v2 quotes? it this collection updates at the end of the day, I will have to switch to production at my morning to keep the data aligned.

p.s. There are 200 contracts, and it takes 37 seconds to update all of them (in historical poll mode).

[alexveden]

As I can see the DataSourceHybrid() is used only in one place in V1 (in EXO builder script), but I can't guarantee that some notebooks don't use it. I've added an exception to this class, and it will raise an error when the DataSourceHybrid() ( 'futurebarcol' ) is referred by code.

[alexveden]

Also, you will have to turn off all code which updates 'contract_bars' from text files.

[alexveden]

I've found a minor discrepancy in RFR series, it looks like 1-day shift because the values are absolutely equal. Could you advise what to do with this?

[spickering-git]

@alexveden go ahead and switch to the tmldb_v2.contracts_bars collection. I am turning off all related CQG updating.

I've found a minor discrepancy in RFR series, it looks like 1-day shift because the values are absolutely equal. Could you advise what to do with this?

This was a legacy artifact that we put into the data collection. Don't worry about it and going forward the days will no longer be shifted.

Also regarding RFR. Could you add a field in the tmqr2.quotes_riskfreerate collection that is just the latest RFR value. When calling the tmqr2.quotes_riskfreerate from the realtime code that is not in python I can't unpickle the rfr_series object. If you just had a field, current_rfr, I could get the latest value which is all the realtime needs.

IQFeed monitoring

When we monitor this page https://10.0.1.2:8889/notebooks/data/IQFeed%20data%20monitoring.ipynb and we see a time_diff_now that looks out of sync

What should the protocol be to bring the data up to date?

[alexveden]

Also regarding RFR. Could you add a field in the tmqr2.quotes_riskfreerate collection that is just the latest RFR value. When calling the tmqr2.quotes_riskfreerate from the realtime code that is not in python I can't unpickle the rfr_series object. If you just had a field, current_rfr, I could get the latest value which is all the realtime needs.

ok

What should the protocol be to bring the data up to date?

I think that the data is up-to-date. My investigation showed:

1. All 24h globex products like ES, CL are up-to-date (10 minutes delays)
2. Some products are delayed, to NOW but they have the last quote which corresponds to the last quotes returned by IQFeed at the moment. For example:

IQ: 2017-12-28 12:40:00-08:00 | QHGF18 Logs (Note the timezone offsets):

[alexveden]

I think that only if all of the asset quotes would be delayed only then we can say that IQFeed has some problems.

[alexveden]

@spickering-git you also have to add ticker EU.$DEFAULT$ to 'asset_info' collection to maintain default asset settings for all tickers in EU market namespace.

[spickering-git]

@alexveden I'm not exactly clear on how you planned the market should be set up. Is it related to currency? Should we set something up for each exchange or should it be more general, (like EU for Europe or the currency Euro). The US includes CME globex and ICE US etc. The EU includes Eurex, Euronext and ICE Europe that are in priced in Euros or Swiss Francs.

[alexveden]

I'm not exactly clear on how you planned the market should be set up. Is it related to currency?

No, the framework is currency agnostic.

Default settings are linked with default trading session, quotes sources, point values etc.

{
    "_id" : ObjectId("58afed10cbf5ba39d96adeb5"),
    "instrument" : "US.$DEFAULT$",
    "market" : "US",
    "rollover_days_before" : 2,
    "rollover_days_before_options" : 2,
    "futures_months" : [ 
        1, 
        2, 
        3, 
        4, 
        5, 
        6, 
        7, 
        8, 
        9, 
        10, 
        11, 
        12
    ],
    "trading_session" : [ 
        {
            "dt" : ISODate("1900-01-01T00:00:00.000Z"),
            "start" : "00:32",
            "decision" : "10:40",
            "execution" : "10:45"
        }
    ],
    "timezone" : "US/Pacific",
    "ticksize" : 1.0,
    "tickvalue" : 1.0,
    "ticksize_options" : 1.0,
    "tickvalue_options" : 1.0,
    "data_futures_src" : "quotes_intraday",
    "data_options_src" : "quotes_options_eod",
    "data_options_use_prev_date" : true
}

[spickering-git]

@alexveden Had an issue with iqfeed which we already discussed. Had to remove EU.$DEFAULT$ and adjust the symbol on an instrument to stop python code from crashing. Then on restart iqfeed script had to loop 11 times before restarting and able to collect data. IQFeedDataFeed.log

[alexveden]

@spickering-git

Ok, I see what was the reason for such error and can fix it. However, we must populate asset index with new contract metadata for EU market to make it properly work. I can implement this in IQFeed Project Milestone 2.

[spickering-git]

@alexveden Can you have a look at this log.

Generated by run_iqfeed_updates.py --live_n_futures=6 running on the windows vm but its throwing a lot of errors.

I have turned it off on the windows vm now and turned it back on in linux.

IQFeedDataFeed.txt

[alexveden]

Have you tried to clone this branch: 'origin/iq_v1_collection' or master? The 'origin/iq_v1_collection' is an actual branch for iqfeed.

I've just pushed some tweak to the 'origin/iq_v1_collection' which will help to catch this issue in the log file. Could you please, try to run the code on windows vm later again?

[spickering-git]

@alexveden yes, it is cloned off branch 'origin/iq_v1_collection' I pulled your updates and ran again on windows VM and it is still throwing lots of error. I'm not sure if it is related to holiday. But I turned it off on windows VM and turned on linux version again. Can you have a look at log again please? IQFeedDataFeed.log

[alexveden]

I'm not sure if it is related to holiday. But I turned it off on windows VM and turned on linux version again.

These errors are related to bar['time'] type returned by PyIQFeed lib. It this possible to allow me access to Windows VM via RDP and Pycharm on it to run this code in the debugger remotely?