Validate FIX repeating groups handling

[salsferrazza]

fix2json doesn't have any special handling of FIX tagval repeating groups; the proper handling would obviously be nesting the objects within the JSON interpretation as represented by the incoming FIX message file.

Need to source (or synthesize) example message file specimens inclusive of repeating groups.

http://fixwiki.org/fixwiki/NoTradingSessions is an example of a repeating field with "No" interpreted as an abbreviation for "Number".

[salsferrazza]

Repeating groups not handled correctly: only the last group member makes it through to the manufactured JSON.

The group will be somewhat liberally interpreted by fix2json.

For example, for a repeating group such as: NoMDEntries; the 'No' prefix will be stripped out, and the remaining characters will be used to name the property of the resultant JSON that will hold the group array.

For example:

{
    "BodyLength": "3972",
    "CheckSum": "117",
    "SecurityIDSource": "EXCHANGE SYMBOL",
    "MsgSeqNum": "1950185",
    "MsgType": "MARKETDATAINCREMENTALREFRESH",
    "SecurityID": "100696",
    "SenderCompID": "CME",
    "SendingTime": "20130715164359550",
    "TradeDate": "20130715",
    "RptSeq": "1603",
    "SecurityDesc": "ESV3 P1220",
    "MDEntries": [
        {
            "MDEntryType": "BID",
            "MDEntryPx": "80",
            "MDEntrySize": "980",
            "MDEntryTime": "164359000",
            "MDUpdateAction": "CHANGE"
        }
    ],
    "TradingSessionID": "HALFDAY",
    "NumberOfOrders": "4",
    "MDPriceLevel": "3",
    "ApplVerID": "FIX50SP2"
}

[salsferrazza]

Wondering if the original "NoMDEntries" in the fictitious output specimen above should be preserved.

[salsferrazza]

Experimentally committed into master, need to secure FIX specimen representative of nested repeating groups for further testing.

[jlroo]

This would be a great feature, I am trying to accomplished the same with a hive script ( nested fix table in hive ). I cloned the repository and placed it at the node_modules folder and I am getting this error, the npm installed version works fine.

[jlroo@fsba fix2json2]# ./fix2json.js
module.js:340
    throw err;
          ^
Error: Cannot find module 'xpath'
    at Function.Module._resolveFilename (module.js:338:15)
    at Function.Module._load (module.js:280:25)
    at Module.require (module.js:364:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/usr/lib/node_modules/fix2json2/fix2json.js:4:13)
    at Module._compile (module.js:456:26)
    at Object.Module._extensions..js (module.js:474:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:312:12)
    at Function.Module.runMain (module.js:497:10)

[salsferrazza]

Hi @jlroo, can you try cloning again then doing an npm install? Just pushed the feature.

[salsferrazza]

Example output as of v0.3.0:

{ "ApplVerID": "FIX50SP2", "BodyLength": "364", "MsgType": "MARKETDATAINCREMENTALREFRESH", "SenderCompID": "CME", "MsgSeqNum": "1408768", "SendingTime": "20130715150258936", "TradeDate": "20130715", "NoMDEntries": "3", "MDEntries": [ { "MDUpdateAction": "DELETE", "SecurityIDSource": "EXCHANGE SYMBOL", "SecurityID": "443979", "RptSeq": "17931", "SecurityDesc": "ESQ3 C1730", "MDEntryType": "BID", "MDEntryPx": "525", "MDEntrySize": "100", "MDEntryTime": "150258000", "TradingSessionID": "HALFDAY", "NumberOfOrders": "1", "MDPriceLevel": "1" }, { "MDUpdateAction": "CHANGE", "SecurityIDSource": "EXCHANGE SYMBOL", "SecurityID": "443979", "RptSeq": "17932", "SecurityDesc": "ESQ3 C1730", "MDEntryType": "BID", "MDEntryPx": "500", "MDEntrySize": "2663", "MDEntryTime": "150258000", "TradingSessionID": "HALFDAY", "NumberOfOrders": "15", "MDPriceLevel": "1" }, { "MDUpdateAction": "NEW", "SecurityIDSource": "EXCHANGE SYMBOL", "SecurityID": "443979", "RptSeq": "17933", "SecurityDesc": "ESQ3 C1730", "MDEntryType": "BID", "MDEntryPx": "490", "MDEntrySize": "116", "MDEntryTime": "150258000", "TradingSessionID": "HALFDAY", "NumberOfOrders": "2", "MDPriceLevel": "3" } ], "CheckSum": "063" }

[jlroo]

Thanks for following up on this so quick @salsferrazza . I am still getting an error when I clone the repo, but v0.3.0 now is creating some nesting json that isn't quite right, it is only showing the last message. Here is what I am getting:

        {
            "ApplVerID": "FIX50SP2",
            "BodyLength": "136",
            "MsgType": "MARKETDATAINCREMENTALREFRESH",
            "SenderCompID": "CME",
            "MsgSeqNum": "578",
            "SendingTime": "20130714180133577",
            "TradeDate": "20130715",
            "NoMDEntries": "1",
            "MDEntries": [
            {
                "MDUpdateAction": "NEW"
            }
            ],
            "SecurityID": "111473",
            "SettlDate": "20130712",
            "RptSeq": "1",
            "SecurityDesc": "ESU4",
            "MDEntryType": "SETTLEMENT PRICE",
            "MDEntryPx": "164350",
            "MDEntryTime": "180133000",
            "CheckSum": "248",
            "SecurityIDSource": "EXCHANGE SYMBOL"
        }
        {
            "ApplVerID": "FIX50SP2",
            "BodyLength": "1417",
            "MsgType": "MARKETDATAINCREMENTALREFRESH",
            "SenderCompID": "CME",
            "MsgSeqNum": "731",
            "SendingTime": "20130714190005961",
            "TradeDate": "20130710",
            "NoMDEntries": "15",
            "MDEntries": [
            {
                "MDUpdateAction": "NEW"
            }
            ],
            "SecurityID": "28112",
            "RptSeq": "16",
            "SecurityDesc": "ESZ3",
            "MDEntryType": "OFFER",
            "MDEntryPx": "172500",
            "MDEntrySize": "1",
            "MDEntryTime": "190005000",
            "TradingSessionID": "0",
            "NumberOfOrders": "1",
            "MDPriceLevel": "5",
            "MDUpdateAction": "NEW",
            "SecurityIDSource": "EXCHANGE SYMBOL",
            "CheckSum": "029"
        }

Here is the FIX message, edited for clarity:

    1128=9^A9=136^A35=X^A49=CME^A34=578^A52=20130714180133577^A75=20130715^A268=1^A279=0^A22=8^A48=111473
        ^A64=20130712^A83=1^A107=ESU4^A269=6^A270=164350^A273=180133000
    ^A10=248^A

    1128=9^A9=1417^A35=X^A49=CME^A34=731^A52=20130714190005961^A75=20130710^A268=15^A279=0^A22=8^A48=28112
        ^A83=2^A107=ESZ3^A269=0^A270=166000^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=1^A279=0^A22=8^A48=28112^A83=3^A107=ESZ3^A269=0^A270=165675^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=2^A279=0^A22=8^A48=28112^A83=4^A107=ESZ3^A269=0^A270=165500^A271=2^A273=190005000^A336=0^A346=1
            ^A1023=3^A279=0^A22=8^A48=28112^A83=5^A107=ESZ3^A269=0^A270=165375^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=4^A279=0^A22=8^A48=28112^A83=6^A107=ESZ3^A269=0^A270=165025^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=5^A279=0^A22=8^A48=28112^A83=7^A107=ESZ3^A269=0^A270=164425^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=6^A279=0^A22=8^A48=28112^A83=8^A107=ESZ3^A269=0^A270=164075^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=7^A279=0^A22=8^A48=28112^A83=9^A107=ESZ3^A269=0^A270=163000^A271=2^A273=190005000^A336=0^A346=1
            ^A1023=8^A279=0^A22=8^A48=28112^A83=10^A107=ESZ3^A269=0^A270=162675^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=9^A279=0^A22=8^A48=28112^A83=11^A107=ESZ3^A269=0^A270=162400^A271=3^A273=190005000^A336=0^A346=1
            ^A1023=10^A279=0^A22=8^A48=28112^A83=12^A107=ESZ3^A269=1^A270=167000^A271=2^A273=190005000^A336=0^A346=1
            ^A1023=1^A279=0^A22=8^A48=28112^A83=13^A107=ESZ3^A269=1^A270=167500^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=2^A279=0^A22=8^A48=28112^A83=14^A107=ESZ3^A269=1^A270=167925^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=3^A279=0^A22=8^A48=28112^A83=15^A107=ESZ3^A269=1^A270=170000^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=4^A279=0^A22=8^A48=28112^A83=16^A107=ESZ3^A269=1^A270=172500^A271=1^A273=190005000^A336=0^A346=1
            ^A1023=5
    ^A10=029^A

Please let me know is there is anything that I can do to help.

[salsferrazza]

Hi @jlroo, so this is an artifact of how repeating groups are determined to be at the end of the group; it actually checks the data dictionary for the list of valid fields for a particular class of group members, then if it sees a tag that doesn't belong to the group currently being processed, it ends the group.

It looks like you are trying to parse CME market data refreshes. I actually tested repeating groups with a similar data source, and discovered that there are some fields in the NoMDEntries group that aren't part of the standard FIX50SP2 dictionary. I've actually made the changes to the data dictionary to accommodate three tags that are in the data but not in the dictionary: SecurityIDSource, SecurityDesc and SecurityID. These modifications have been committed to the repo as FIX50SP2-customized.xml, so if you call fix2json using that dictionary instead, you'll likely see more desirable output. For example:

cat MDFF_CME_20130714-20130715_7818_0 |head -1 | fix2json -p ~/Desktop/fix2json/dict/FIX50SP2-customized.xml |more
{
    "ApplVerID": "FIX50SP2",
    "BodyLength": "3972",
    "MsgType": "MARKETDATAINCREMENTALREFRESH",
    "SenderCompID": "CME",
    "MsgSeqNum": "1950185",
    "SendingTime": "20130715164359550",
    "TradeDate": "20130715",
    "NoMDEntries": "39",
    "MDEntries": [
        {
            "MDUpdateAction": "CHANGE",
            "SecurityIDSource": "EXCHANGE SYMBOL",
            "SecurityID": "259057",
            "RptSeq": "47433",
            "SecurityDesc": "ESU3 P1675",
            "MDEntryType": "OFFER",
            "MDEntryPx": "3800",
            "MDEntrySize": "579",
            "MDEntryTime": "164359000",
            "TradingSessionID": "HALFDAY",
            "NumberOfOrders": "3",
            "MDPriceLevel": "1"
        },
        {
            "MDUpdateAction": "CHANGE",
            "SecurityIDSource": "EXCHANGE SYMBOL",
            "SecurityID": "259057",
            "RptSeq": "47434",
            "SecurityDesc": "ESU3 P1675",
            "MDEntryType": "OFFER",
            "MDEntryPx": "3825",
            "MDEntrySize": "1047",
            "MDEntryTime": "164359000",
            "TradingSessionID": "HALFDAY",
            "NumberOfOrders": "6",
            "MDPriceLevel": "2"
        },
        {
            "MDUpdateAction": "CHANGE",
            "SecurityIDSource": "EXCHANGE SYMBOL",
            "SecurityID": "584604",
            "RptSeq": "48289",
            "SecurityDesc": "ESU3 P1680",
            "MDEntryType": "BID",
            "MDEntryPx": "3950",
:
...

Please give that a shot and let me know how it works out for you.

Thanks!

Sal

[salsferrazza]

Oh, and after you clone the repo you'll have to cd into the clone directory and run npm install so that fix2json can pick up its prerequisites.

$ node ./fix2json.js
module.js:338
    throw err;
          ^
Error: Cannot find module 'xpath'
    at Function.Module._resolveFilename (module.js:336:15)
    at Function.Module._load (module.js:278:25)
    at Module.require (module.js:365:17)
    at require (module.js:384:17)
    at Object.<anonymous> (/mnt/replica/home/salzie/src/fix2json/fix2json.js:4:13)
    at Module._compile (module.js:460:26)
    at Object.Module._extensions..js (module.js:478:10)
    at Module.load (module.js:355:32)
    at Function.Module._load (module.js:310:12)
    at Function.Module.runMain (module.js:501:10)
$ npm install
string_decoder@0.10.31 node_modules/string_decoder

node-xml@1.0.2 node_modules/node-xml

underscore@1.7.0 node_modules/underscore

xmldom@0.1.19 node_modules/xmldom

xpath@0.0.9 node_modules/xpath
$ node ./fix2json.js
Usage: fix2json [-p] <data dictionary xml file> [path to FIX message file]

fix2json will use standard input in the absence of a message file.

[jlroo]

Hi Sal,

Thank you so much for the customized format. I gave it a try, and it worked like a charm. This is my first time working with FIX, and there no readily available resources. You guys have been really helpful. I will definitely recommend your repo - this is the only resource that is up to date, and well supported.

Thank you again,

Jose

[salsferrazza]

Thanks for the kind words and feedback, Jose! Don't hesitate to let us know if you have any suggestions or questions as you get to using the tool a bit more.