Closed jlroo closed 8 years ago
salsferrazza
commented
8 years ago Hi jlroo,
To be clear, are you referring to the topmost tags with unresolved tag mnemonics?
"5796": "20130712",
"9787": "0.01",
"9850": "0",If this is the case, then adjusting the data dictionary XML is relatively straightforward.
Do you have the relevant CME FIX spec for the data file handy?
salsferrazza
commented
8 years ago Found references to those tags here: http://www.cmegroup.com/confluence/display/EPICSANDBOX/Market+Data+-+Security+Definition
5796 TradingReferenceDate LocalMktDate(8) 9787 DisplayFactor Float(20) 9850 MinCabPrice Price(20)
Patching the dictionary manually is a two step process:
Step 1
open the dictionary file up in your text (or XML) editor of choice and navigate to where the Security Definition message type fields are defined:
<message name='SecurityDefinition' msgcat='app' msgtype='d'>
<component name='ApplicationSequenceControl' required='N' />
<field name='SecurityReportID' required='N' />
<field name='ClearingBusinessDate' required='N' />
<field name='SecurityReqID' required='N' />
<field name='SecurityResponseID' required='N' />
...You would add these three lines under the message tag and above the first field tag:
<field name='TradingReferenceDate' required='N' />
<field name='DisplayFactor' required='N' />
<field name='MinCabPrice' required='N' />The resulting section for SecurityDefinition would then read as:
<message name='SecurityDefinition' msgcat='app' msgtype='d'>
<component name='ApplicationSequenceControl' required='N' />
<field name='TradingReferenceDate' required='N' />
<field name='DisplayFactor' required='N' />
<field name='MinCabPrice' required='N' />
<field name='SecurityReportID' required='N' />
<field name='ClearingBusinessDate' required='N' />
<field name='SecurityReqID' required='N' />
...Step 2
In the same file, navigate to where the fields are defined.. search for </fields>:
...
<field number='1617' name='StreamAsgnType' type='INT'>
<value enum='1' description='ASSIGNMENT' />
<value enum='2' description='REJECTED' />
<value enum='3' description='TERMINATE_UNASSIGN' />
</field>
</fields>
</fix>Above the closing </fields> tag, add these three lines:
<field number='5796' name='TradingReferenceDate' type='LOCALMKTDATE'/>
<field number='9787' name='DisplayFactor' type='FLOAT' />
<field number='9850' name='MinCabPrice' type='PRICE' />The resultant section will then read as:
...
<field number='1504' name='RelSymTransactTime' type='UTCTIMESTAMP' />
<field number='1617' name='StreamAsgnType' type='INT'>
<value enum='1' description='ASSIGNMENT' />
<value enum='2' description='REJECTED' />
<value enum='3' description='TERMINATE_UNASSIGN' />
</field>
<field number='5796' name='TradingReferenceDate' type='LOCALMKTDATE'/>
<field number='9787' name='DisplayFactor' type='FLOAT' />
<field number='9850' name='MinCabPrice' type='PRICE' />
</fields>
</fix>Save the data dictionary file and let fix2json know you'd like it to be used:
grep -h 9787 secdefnewrelease.dat | head -1 | fix2json -p FIX50SP2-customized.xml | grep MinCabPrice
"MinCabPrice": "0.02",The same process can be applied for any other customizations. Please try and report back.
Thanks!
Sal
P.S. https://github.com/SunGard-Labs/fix2json/issues/2 hopes to address the patching of stock dictionaries using XML merges, so that patches can be stored separately from the original files.
jlroo
commented
8 years ago It worked great thank you! I see how XML merges will be really helpful to keep track of stock dictionaries. Can you expand more on how to implement XML merges? Thank you again for your help.
salsferrazza
commented
8 years ago From a CLI perspective, right now you tell fix2json which dictionary you want to use on the file system, he reads it and interprets the FIX input files according to the specific dictionary that was passed in. In other words, there is a one-to-one relationship between the dictionary source data and what an instance of fix2json can address. This keeps dictionary management simple (by eliminating it), but without much flexibility or sophistication.
One could imagine a dictionary representation that instead of a single file, uses a sub-hierarchy of a filesystem to represent the logical contents of a dictionary according to a convention. One possibility might be to organize the dictionary contents as such:
fix2json/
fix2json/dict/
fix2json/dict/FIX42
fix2json/dict/FIX42/FIX42.xml # <- stock QuickFIX XML dictionary for FIX 4.2
fix2json/dict/FIX50SP2/FIX50SP2.xml # <- stock QuickFIX XML dictionary for FIX 5.0 SP2
fix2json/dict/FIX50SP2/cme/
fix2json/dict/FIX50SP2/cme/secdef/
fix2json/dict/FIX50SP2/cme/secdef/patch.xmlThe stock dictionaries would be identical to the ones that ship with the fix2json npm, but instead of being in the dict/ directory as complete XML files, a sub-directory per FIX version will exist, and the dictionaries will reside there.
Then a directory for each customization can be created, so that one may easily catalog the origins of the customizations. In the example representation above, the contents of the patch.xml file corresponding to the specific customization of this issue would be:
<fix major='5' type='FIX' servicepack='2' minor='0'>
<header />
<messages>
<message name='SecurityDefinition' msgcat='app' msgtype='d'>
<field name='TradingReferenceDate' required='N' />
<field name='DisplayFactor' required='N' />
<field name='MinCabPrice' required='N' />
</message>
</messages>
<fields>
<field number='5796' name='TradingReferenceDate' type='LOCALMKTDATE'/>
<field number='9787' name='DisplayFactor' type='FLOAT' />
<field number='9850' name='MinCabPrice' type='PRICE' />
</fields>
<fix>Then from a CLI parameter standpoint, instead of specifying a file that contains the entire dictionary, one simply specifies a version of FIX to target, e.g.:
fix2json FIX50SP2 fix50sp2file.txtfix2json would then traverse the dict/FIX50SP2 directory and combine (via XML merge) the stock dictionary for the version and all physical customization fragments within the dict subdirectories into a single logical data dictionary to be used during interpretation of the FIX stream.
Sure beats opening up a massive XML file to perform surgery and would certainly be less error-prone. If you have any suggestions or alternatives to the method proposed above, we'd love to hear them.
Glad those customizations worked out, and thank you for helping fix2json put some more thought into dictionary management!
jlroo
commented
8 years ago This is really helpful! Thank you for your help.
Hi guys, thank you for developing this tool it has been really helpful on my research. I am having a issue with a new data set from the CME that has custom tags. How can I add these tags to the custom dictionary? Here is part of the FIX msg and the same decoded msg:
CME Fix message
Decoded using fix2json