search

FIXTradingCommunity / orchestrations

Service offerings expressed with Orchestra
Apache License 2.0
11 stars 8 forks source link

FixRepository42.xml QuoteRequest is incorrect #12

Closed GaryHughes closed 3 years ago

GaryHughes commented 4 years ago

I've been replacing a code generator that previously used the repository with a version that uses the orchestra standard and I'm using the files published here has a reference. Some tests I had broke and upon investigation it appears that the FixRepository42.xml file has an incomplete definition for QuoteRequest, I first noticed this because there was no PrevClosePx field. I've attached a dump of the structure I build below for the 4.2 and 4.2 files.

QuoteRequest42.txt QuoteRequest44.txt

Here is the structure from the 4.2 file.

MsgType = R Number of related symbols in Request In some markets it is the practice to request quotes from brokers prior to placement of an order. The quote request message is used for this purpose.
kleihan commented 4 years ago

@GaryHughes you are referencing 2151 which is the StrmAsgnRptInstrmtGrp component. I would have expected the QuotReqGrp component which has id=2045 and contains the PrevClosePx(140) field. Maybe it is a coincidence but both components use the NumInGroup field NoRelatedSym(146). Could that be the reason?

Your 4.2 dump also does not contain any instrument component fields like the FIX 4.4 dump does.

donmendelson commented 4 years ago

@kleihan, I believe you're on the right track. In the Unified Repository file for FIX 4.2, repeating groups have no identification other than the tag of the NumInGroup.

Comment in unified2orchestra script:

Find component for this repeating group. If FIX 4.4 or later, it will be the parent element. Otherwise, search for last component with matching NumInGroup id.

I suggest a manual fix for this.

GaryHughes commented 4 years ago

@kleihan you are correct about the dump missing the instrument fields and that is the problem :) If you look at the extract I included above for QuoteRequest it only contains the 1001 (standard header), 1002 (standard trailer), 131 (QuoteReqID), and 2151 (StrmAsgnRptInstrmtGrp). 2151 only contains fieldRefs.

@donmendelson I don't understand the NuminGroup comment, am I supposed to treat the as a componentRef? In which case there is no matching component, the only other reference to 146 in the file is a field.

What is the suggested manual fix? and is that for me to do or you?

I appreciate the help.

donmendelson commented 4 years ago

@GaryHughes there is indeed a flaw in the file, I was just giving an explanation for why it happened. I will fix it; nothing you have to do.

kleihan commented 4 years ago

@donmendelson is the problem limited to QuoteRequest? I see that FIX 4.2 uses NoRelatedSym(146) for 6 different messages. Some of the components are similar/identical (Email, News, MarketDataRequest), others are quite different. The FIX 4.2 dump from @GaryHughes looks like the content of the component in SecurityDefinition(Request), e.g. no instrument fields, just underlying instrument fields.

GaryHughes commented 4 years ago

I haven't reviewed further but it seems likely there are other similar issues, I'm planning on writing a script to do a sanity check of the orchestra against the repository and I'll be happy to share that when it's done.

GaryHughes commented 4 years ago

I've done a rough first cut of an audit script in Python, you can find it here I'll add some proper documentation and there are a lot more tests that can be aded. Here is some sample output, I'll improve it so it gives details on the field mismatches.

$ ./fixaudit.py --orchestration FixRepository42.xml --repository fix_repository_2010_edition_20200402/FIX.4.2/Base Fields Orchestration = 405 Repository = 405 All fields have the same Name and Added values in the repository and the orchestration Messages Orchestration = 46 Repository = 46 The following 12 discrepancies were found message MsgType = B has 65 fields in the repository and 68 fields in the orchestration message MsgType = C has 67 fields in the repository and 70 fields in the orchestration message MsgType = D has 104 fields in the repository and 128 fields in the orchestration message MsgType = E has 119 fields in the repository and 52 fields in the orchestration message MsgType = G has 103 fields in the repository and 127 fields in the orchestration message MsgType = J has 111 fields in the repository and 116 fields in the orchestration message MsgType = R has 63 fields in the repository and 54 fields in the orchestration message MsgType = V has 58 fields in the repository and 60 fields in the orchestration message MsgType = W has 84 fields in the repository and 110 fields in the orchestration message MsgType = Z has 56 fields in the repository and 72 fields in the orchestration message MsgType = b has 81 fields in the repository and 97 fields in the orchestration message MsgType = k has 77 fields in the repository and 85 fields in the orchestration

GaryHughes commented 4 years ago

Ok I've done enough to see the differences now, no guarantees about the correctness yet but I think it's ok. 4.4 looks pretty good, 4.2 appears to have a fair few issues. The repo I've linked above also contains my scripts for dumping the definition of fields and messages from either the repository or an orchestration.

$ ./fixaudit.py --orchestration FixRepository44.xml --repository fix_repository_2010_edition_20200402/FIX.4.4/Base Fields Orchestration = 912 Repository = 912 All fields have the same Name and Added values in the repository and the orchestration Messages Orchestration = 93 Repository = 93 The following 2 discrepancies were found message MsgType = A has 46 fields in the repository and 43 fields in the orchestration message MsgType = A repository has the following fields not in the corresponding orchestration message ['NoMsgTypes', 'MsgDirection', 'RefMsgType']

$ ./fixaudit.py --orchestration FixRepository42.xml --repository fix_repository_2010_edition_20200402/FIX.4.2/Base Fields Orchestration = 405 Repository = 405 All fields have the same Name and Added values in the repository and the orchestration Messages Orchestration = 46 Repository = 46 The following 33 discrepancies were found message MsgType = B has 65 fields in the repository and 68 fields in the orchestration message MsgType = B orchestration has the following fields not in the corresponding repository message ['UnderlyingIDSource', 'UnderlyingIssuer', 'UnderlyingSecurityDesc', 'UnderlyingContractMultiplier', 'UnderlyingCouponRate', 'UnderlyingSecurityExchange', 'Side', 'UnderlyingSymbolSfx', 'UnderlyingMaturityMonthYear', 'UnderlyingSecurityID', 'UnderlyingSecurityType', 'UnderlyingMaturityDay', 'UnderlyingOptAttribute', 'UnderlyingPutOrCall', 'UnderlyingStrikePrice', 'UnderlyingSymbol', 'UnderlyingCurrency', 'RatioQty', 'EncodedUnderlyingIssuerLen', 'EncodedUnderlyingIssuer', 'EncodedUnderlyingSecurityDescLen', 'EncodedUnderlyingSecurityDesc'] message MsgType = B repository has the following fields not in the corresponding orchestration message ['IDSource', 'SecurityType', 'RelatdSym', 'SecurityID', 'SymbolSfx', 'MaturityMonthYear', 'PutOrCall', 'StrikePrice', 'MaturityDay', 'OptAttribute', 'SecurityExchange', 'EncodedIssuerLen', 'EncodedIssuer', 'EncodedSecurityDescLen', 'EncodedSecurityDesc', 'CouponRate', 'ContractMultiplier', 'Issuer', 'SecurityDesc'] message MsgType = C has 67 fields in the repository and 70 fields in the orchestration message MsgType = C orchestration has the following fields not in the corresponding repository message ['UnderlyingIDSource', 'UnderlyingIssuer', 'UnderlyingSecurityDesc', 'UnderlyingContractMultiplier', 'UnderlyingCouponRate', 'UnderlyingSecurityExchange', 'Side', 'UnderlyingSymbolSfx', 'UnderlyingMaturityMonthYear', 'UnderlyingSecurityID', 'UnderlyingSecurityType', 'UnderlyingMaturityDay', 'UnderlyingOptAttribute', 'UnderlyingPutOrCall', 'UnderlyingStrikePrice', 'UnderlyingSymbol', 'UnderlyingCurrency', 'RatioQty', 'EncodedUnderlyingIssuerLen', 'EncodedUnderlyingIssuer', 'EncodedUnderlyingSecurityDescLen', 'EncodedUnderlyingSecurityDesc'] message MsgType = C repository has the following fields not in the corresponding orchestration message ['IDSource', 'SecurityType', 'RelatdSym', 'SecurityID', 'SymbolSfx', 'MaturityMonthYear', 'PutOrCall', 'StrikePrice', 'MaturityDay', 'OptAttribute', 'SecurityExchange', 'EncodedIssuerLen', 'EncodedIssuer', 'EncodedSecurityDescLen', 'EncodedSecurityDesc', 'CouponRate', 'ContractMultiplier', 'Issuer', 'SecurityDesc'] message MsgType = D has 104 fields in the repository and 122 fields in the orchestration message MsgType = D orchestration has the following fields not in the corresponding repository message ['SettlInstMode', 'AllocText', 'NoMiscFees', 'MiscFeeAmt', 'MiscFeeCurr', 'MiscFeeType', 'EncodedAllocTextLen', 'EncodedAllocText', 'AllocPrice', 'NotifyBrokerOfCredit', 'SettlCurrFxRateCalc', 'AllocHandlInst', 'SettlCurrAmt', 'AllocAvgPx', 'AllocNetMoney', 'SettlCurrFxRate', 'BrokerOfCredit', 'AccruedInterestAmt'] message MsgType = E has 119 fields in the repository and 52 fields in the orchestration message MsgType = E orchestration has the following fields not in the corresponding repository message ['AvgPx', 'OrdStatus', 'OrdRejReason', 'CumQty', 'CxlQty', 'LeavesQty'] message MsgType = E repository has the following fields not in the corresponding orchestration message ['Account', 'NoTradingSessions', 'DiscretionInst', 'DiscretionOffset', 'Commission', 'CommType', 'PrevClosePx', 'Currency', 'SideValueInd', 'ExecInst', 'HandlInst', 'IDSource', 'IOIid', 'CashOrderQty', 'SettlInstMode', 'OrderQty', 'SecurityType', 'OrdType', 'EffectiveTime', 'GTBookingInst', 'Price', 'Rule80A', 'SecurityID', 'ExpireDate', 'Side', 'Symbol', 'ClearingFirm', 'ClearingAccount', 'TimeInForce', 'TransactTime', 'SettlmntTyp', 'FutSettDate', 'SymbolSfx', 'OrderQty2', 'ListSeqNo', 'FutSettDate2', 'MaturityMonthYear', 'PutOrCall', 'StrikePrice', 'CoveredOrUncovered', 'ExecBroker', 'OpenClose', 'NoAllocs', 'AllocAccount', 'AllocShares', 'ProcessCode', 'CustomerOrFirm', 'MaturityDay', 'OptAttribute', 'SecurityExchange', 'MaxShow', 'PegDifference', 'TradingSessionID', 'EncodedIssuerLen', 'EncodedIssuer', 'EncodedSecurityDescLen', 'CouponRate', 'EncodedSecurityDesc', 'StopPx', 'ExDestination', 'ComplianceID', 'ContractMultiplier', 'SolicitedFlag', 'Issuer', 'SecurityDesc', 'ClientID', 'MinQty', 'MaxFloor', 'LocateReqd', 'QuoteID', 'SettlCurrency', 'ForexReq', 'ExpireTime'] message MsgType = G has 103 fields in the repository and 122 fields in the orchestration message MsgType = G orchestration has the following fields not in the corresponding repository message ['NoMiscFees', 'MiscFeeAmt', 'MiscFeeCurr', 'MiscFeeType', 'AllocAvgPx', 'AllocNetMoney', 'SettlCurrFxRate', 'SettlCurrFxRateCalc', 'AccruedInterestAmt', 'SettlInstMode', 'AllocText', 'NotifyBrokerOfCredit', 'ProcessCode', 'AllocHandlInst', 'BrokerOfCredit', 'EncodedAllocTextLen', 'EncodedAllocText', 'AllocPrice', 'SettlCurrAmt'] message MsgType = J has 111 fields in the repository and 112 fields in the orchestration message MsgType = J orchestration has the following fields not in the corresponding repository message ['OrdRejReason', 'OrdStatus', 'CumQty', 'CxlQty', 'LeavesQty'] message MsgType = J repository has the following fields not in the corresponding orchestration message ['WaveNo', 'ListID', 'OrderID', 'SecondaryOrderID'] message MsgType = R has 63 fields in the repository and 54 fields in the orchestration message MsgType = R orchestration has the following fields not in the corresponding repository message ['UnderlyingIDSource', 'UnderlyingIssuer', 'UnderlyingSecurityDesc', 'UnderlyingContractMultiplier', 'UnderlyingCouponRate', 'UnderlyingSecurityExchange', 'UnderlyingSecurityID', 'UnderlyingSymbolSfx', 'UnderlyingSecurityType', 'UnderlyingMaturityMonthYear', 'UnderlyingMaturityDay', 'UnderlyingPutOrCall', 'UnderlyingOptAttribute', 'UnderlyingStrikePrice', 'UnderlyingSymbol', 'UnderlyingCurrency', 'RatioQty', 'EncodedUnderlyingIssuerLen', 'EncodedUnderlyingIssuer', 'EncodedUnderlyingSecurityDescLen', 'EncodedUnderlyingSecurityDesc'] message MsgType = R repository has the following fields not in the corresponding orchestration message ['PrevClosePx', 'Currency', 'IDSource', 'OrderQty', 'SecurityType', 'OrdType', 'QuoteRequestType', 'SecurityID', 'Symbol', 'TransactTime', 'FutSettDate', 'SymbolSfx', 'FutSettDate2', 'OrderQty2', 'MaturityMonthYear', 'PutOrCall', 'StrikePrice', 'MaturityDay', 'OptAttribute', 'SecurityExchange', 'TradingSessionID', 'EncodedIssuerLen', 'EncodedIssuer', 'EncodedSecurityDescLen', 'CouponRate', 'EncodedSecurityDesc', 'ContractMultiplier', 'Issuer', 'SecurityDesc', 'ExpireTime'] message MsgType = V has 58 fields in the repository and 60 fields in the orchestration message MsgType = V orchestration has the following fields not in the corresponding repository message ['UnderlyingIDSource', 'UnderlyingIssuer', 'UnderlyingCouponRate', 'UnderlyingContractMultiplier', 'UnderlyingSecurityExchange', 'Side', 'UnderlyingSecurityDesc', 'UnderlyingSymbolSfx', 'UnderlyingMaturityMonthYear', 'UnderlyingSecurityID', 'UnderlyingSecurityType', 'UnderlyingMaturityDay', 'UnderlyingOptAttribute', 'UnderlyingPutOrCall', 'UnderlyingStrikePrice', 'UnderlyingSymbol', 'UnderlyingCurrency', 'RatioQty', 'EncodedUnderlyingIssuerLen', 'EncodedUnderlyingIssuer', 'EncodedUnderlyingSecurityDescLen', 'EncodedUnderlyingSecurityDesc'] message MsgType = V repository has the following fields not in the corresponding orchestration message ['IDSource', 'SecurityType', 'SecurityID', 'Symbol', 'SymbolSfx', 'MaturityMonthYear', 'PutOrCall', 'StrikePrice', 'MaturityDay', 'OptAttribute', 'SecurityExchange', 'TradingSessionID', 'EncodedIssuerLen', 'EncodedIssuer', 'EncodedSecurityDescLen', 'EncodedSecurityDesc', 'CouponRate', 'ContractMultiplier', 'Issuer', 'SecurityDesc'] message MsgType = W has 84 fields in the repository and 88 fields in the orchestration message MsgType = W orchestration has the following fields not in the corresponding repository message ['MDEntryRefID', 'DeleteReason', 'MDEntryID', 'MDUpdateAction'] message MsgType = Z has 56 fields in the repository and 71 fields in the orchestration message MsgType = Z orchestration has the following fields not in the corresponding repository message ['FutSettDate', 'FutSettDate2', 'OrderQty2', 'BidPx', 'OfferPx', 'BidSize', 'OfferSize', 'OrdType', 'QuoteEntryID', 'OfferSpotRate', 'TransactTime', 'Currency', 'BidSpotRate', 'BidForwardPoints', 'ValidUntilTime', 'OfferForwardPoints'] message MsgType = Z repository has the following fields not in the corresponding orchestration message ['UnderlyingSymbol'] message MsgType = b has 81 fields in the repository and 96 fields in the orchestration message MsgType = b orchestration has the following fields not in the corresponding repository message ['FutSettDate', 'OrderQty2', 'FutSettDate2', 'BidPx', 'OfferPx', 'BidSize', 'OfferSize', 'OrdType', 'OfferSpotRate', 'Currency', 'QuoteSetValidUntilTime', 'BidSpotRate', 'TransactTime', 'BidForwardPoints', 'ValidUntilTime', 'OfferForwardPoints'] message MsgType = b repository has the following fields not in the corresponding orchestration message ['QuoteEntryRejectReason'] message MsgType = k has 77 fields in the repository and 81 fields in the orchestration message MsgType = k orchestration has the following fields not in the corresponding repository message ['Country', 'PriceType', 'Commission', 'CommType', 'Price'] message MsgType = k repository has the following fields not in the corresponding orchestration message ['Account']

donmendelson commented 4 years ago

@GaryHughes would you be willing to share your audit script? We would like to use it, or some form of it, for QA and to guide the correct solution for FIX 4.2.

GaryHughes commented 4 years ago

Sure,you can find it here https://github.com/GaryHughes/fixorchestra

I plan on doing some docs for it today but it's pretty simple to use if you're at all familiar with Python and you can run with --help to see the options.

donmendelson commented 4 years ago

@GaryHughes Thanks!

GaryHughes commented 4 years ago

Please feel free to raise issues for any problems or features you'd like added.

GaryHughes commented 4 years ago

HI do you have any updates on this? I'd love a new 4.2 orchestration so I can complete some tasks.

Thanks.

GaryHughes commented 4 years ago

I've ended up writing my own repository to orchestration conversion script that fixes the problems I've been having. it's not complete yet (some missing attributes etc) but it converts all the message structure. If you're interested you can see it here. This will also correct some errors in the source repositories such as missing data types in the 4.x versions so the resulting orchestration is internally consistent.

donmendelson commented 4 years ago

For FIX 4.2, our goal is to have a solution that extracts the in-line repeating groups from Repository and defines them as <group> in Orchestra format, using names and IDs derived from later versions of FIX. We expect to post this by the end of the month.

donmendelson commented 4 years ago

Checking in what I hope is a correct solution, but asking @GaryHughes and @kleihan to verify.

One thing I discovered is that there is a repeating group in SecurityDefinition [type 'd'] with NumInGroup=146. When I look at the same message type in FIX 4.4, it contains repeating group UndInstrmtGrp that appears to have the same contents, but the NumInGroup changed to 711. I mention this because it may be related to the original problem with QuoteRequest which has a different repeating group with NumInGroup=146.

GaryHughes commented 4 years ago

Thanks @donmendelson, one immediate problem, group 2043 in the following groupRef is not defined. This broke my validation script so I'll make it more robust and let you know if I see anything else.

The number of quotes for this Symbol (QuoteSet) that follow in this message. ** Nested Repeating Group follows **
GaryHughes commented 4 years ago

@donmendelson that looks like the only internal consistency issue, barring anything wrong with group 2043. When this is fixed I'll run the comparison against the matching repository.

% ./fixaudit.py --orchestration ../../orchestrations/FIX\ Standard/FixRepository42.xml Validating orchestration All data types referenced by fields are defined All referenced fields are defined message MsgType=i references group id=2049 that contains a reference to group id=2043 that is not defined All referenced components are defined

kleihan commented 4 years ago

Component 2043 is QuotEntryGrp which is nested inside component 2049 (QuotSetGrp). Maybe the only instance in FIX 4.2 of nesting to the second level.

kleihan commented 4 years ago

@donmendelson SecurityDefinition(35=d) apparently went through major changes more than once from FIX 4.2 to 4.3 and then again to 4.4. Fields for underlying instruments were present in 4.2, absent in 4.3 and back again in 4.4, albeit as part of different repeating groups. Also, Side(54) was part of that repeating group in FIX 4.2 and no longer part of it in FIX 4.4. RatioQty(319) was also part of the group in FIX 4.2 and "vanished" with FIX 4.3. In short, the group that existed in FIX 4.2 is unique to FIX 4.2 and was removed as of FIX 4.3. UndInstrmtGrp may look similar in FIX 4.4 but should be viewed as a different group.

In general it is not an issue to have a NumInGroup field like 146 with different content, i.e. the NumInGroup field does not identify a repeating group. The repeating groups did not have names in FIX 4.2.

donmendelson commented 4 years ago

It looks like the error is in the definition of QuotEntryGrp. It should be id="2043" instead of 2042.

GaryHughes commented 4 years ago

This doesn't appear to be fixed? There is still a groupRef to 2042 in FixRepository42.xml

kleihan commented 4 years ago

I believe @GaryHughes is right but I see the error as a different one. Both the MassQuote(35=i) and the QuoteAcknowledgment(35=b) (later renamed to MassQuoteAck) have a repeating group of quote sets that have a nested repeating group of quote entries. The groupRef 2042 is for the nested entries in the ack message. At first glance the issue seems to be that this group (QuoteEntryAckGrp) is not defined in the FIX 4.2 Orchestra file yet. The following are defined:

QuoteSetGrp (groupRef=2049) QuoteSetAckGrp (groupRef=2048) QuoteEntryGrp (groupRef=2043)

QuoteEntryAckGrp groupRef=2042) is a sub- and superset of QuoteEntryGrp (groupRef=2043) as follows:

The FIX 4.2 Orchestra file defines QuoteEntryRejectReason(368) as part of QuoteEntryGrp (groupRef=2043). However, I believe the issue is that the definition of QuoteEntryGrp (groupRef=2043) is actually the definition of QuoteEntryAckGrp (groupRef=2042). The result of my analysis is hence:

1. QuoteEntryAckGrp (groupRef=2042) is defined in terms of content but runs under QuoteEntryGrp (groupRef=2043) 2. QuoteEntryGrp (groupRef=2043) needs to be defined as a new group with its own set of field references

Actions are hence:

GaryHughes commented 3 years ago

This looks good now, thanks!

% ./fixaudit.py --orchestration ../../orchestrations/FIX\ Standard/FixRepository42.xml Validating orchestration All data types referenced by fields are defined All referenced fields are defined All referenced groups are defined All referenced components are defined