iMpact dissector is out of date

[shatteringlass]

First of all, congratulations on this project and thank you so much for the effort you put into this.

While trying to dissect some demo data from ICE iMpact I noticed that there are many unknown messages. This is due to the fact that the message spec for iMpact has now moved to v.1.33 (as opposed to 1.24), in order to conform to new market requirements (including MIFID2).

I tried to put together a changelog for the message spec:

• Updated the description of HasPreOpenVolume in Msgtype ‘g’
• Updated the description of GBP and EUR price in Fixing Lockdown Message
• Updated the description of SettlePriceDateTime in MesssageType ‘C’ from -1 to 0 to reflect actual behavior
• Added new Field StrategyPreference to Message ‘1’
• Added new Field AONAllowed to Message ‘B’
• Added new Field MiFIDRegulatedMarket to ‘B’, ‘p’, ‘R’, ‘’l’, ‘d’, ‘q, ‘U’, ‘9’
• Added Special Field AON to Message ‘D’, ‘E’ in Appedix G
• Added new message Fixing Indicative Price Message
• Added following Fields to Message ‘q’, ‘U’:
• LegStrategyCode, LegRatioQtyNumerator, LegRatioQtyDenominator, LegRatioPriceNumerator, LegRatioPriceDenominator, HedgeStrategyCode, DealPriceDenominator, SettlePriceDenominator, UnitQtyDenominator
• Added following Fields to Message ‘d’, ‘9’: LegStrategyCode, LegRatioQtyNumerator, LegRatioQtyDenominator, LegRatioPriceNumerator, LegRatioPriceDenominator, UnderlyingMarketId, MarketDesc, MaturityYear, MaturityMonth,MaturityDay, DealPriceDenominator, UnitQuantity, NumDecimalsOptionsPrice, AllowOptions, ClearedAlias, AllowImplied, MinPrice, MaxPrice, ProductName, HubAlias, StripName, IsTradable, SettlePriceDenominator, MicCode, UnitQtyDenominator, HedgeOnly, ExchangeSilo, OffExchangeIncrementQtyDenominator, OffExchangeIncrementQty, OffExchangeIncrementPrice,OffExchangeIncrementOptionPrice, ProductID, HubID, StripID, ISIN
• Corrected some fields which were overwritten in Message 9 and d
• Corrected the size of LegRatio fields from 8 to 4
• Removed the offsets after the repeating groups in Message 9
• Added description for LegRatio fields and Leg strategy code
• Updated the description of SettlePriceDateTime in MesssageType ‘C’
• Changed underlyingMarketID to ReservedField in Message 9 and d
• Marked Bilateral Off Exchange Trade type for future removal since this type is not being sent now
• Updated the New Futures Strategy Definition - removed LotSize, IsSpread, IsCrackSpread, PrimaryMarketID, Secondary Market ID, Currency, TickValue, OptionsExpirationYear,OptionsExpirationMonth, OptionsExpirationDay, ReservedField1, IsSerialOptionSupported
• Added New Message Indicative Quote Message
• Added DateTime and SequenceWithinMillis to Delete Order Message
• Added newField TestMarketIndicator to Message ‘B’, ‘p’, ‘q’, ‘U’, ‘d’, ‘9’
• Added BlockDetails to ‘B’,’p’,’q’,’d’
• Added RefSpreadProductId to ‘B’
• Added Missing field BlockDetailLength to ‘B’,’p’,’q’,’d’
• Updated the description of NumBlockDetails in ’q’ and ’d’
• Updated ExtraFlag bit 3 field in Trade Message as isVerticalSplit
• Added ISIN to ‘B’, ‘p’ and renamed the old field to Underlying ISIN in B and d
• Added ModificationTimestamp to Add/Modify Order Message ‘E’
• Added ContractSymbolExtra to ‘U’, ‘q’
• Added RFQQuantity, SecurityType, OptionType, StartDate, EndDate to Indicative Quote message ‘n’
• Updated the description of USD price for Fixing Lockdown Message
• Added Timestamp to Add, Change and Delete Price Level Message
• Updated the description of ValuationDateTime in O
• Updated the description of IsImplied bit field on the trade to reflect actual behavior
• Renamed ISIN to Underlying ISIN in 9 and R
• Updated the description of ValuationDateTime in Option Settlement Price Message (w)
• Removed Indicative Quote Message
• Add pre-close state to Appendix A. Pre-close is still being used for some markets.
• Updated description of HasPreviousDaySettlementPrice and PreviousDaySettlementPrice
• Updates to the Fixing Lockdown Message
• Added links for additional documents for IRS and CDS Swaps in Appendix G
• Added NumberOfExtraLegDefinitions and subsequent Leg definitions to d and q
• Added MarketTransparencyType to B
• Added LegRatioPriceDenominator to q
• Added LegDealSuppressed to q, d, 9, and U
• Updated description of RepurchaseRate, RepurchaseDate, Factor, AccruedPremiumAmt and EventPaymentAmt in B
• Updated the version to match the ICE Impact Feed Multicast Technical Specification
• Updated the valid values for Block Trade Type
• Added ScreenLastTradeDate special field type
• Added Special Field type “ScreenLastTradeDate”
• Added ScreenLastTradeYear, ScreenLastTradeMonth, and ScreenLastTradeDay to ‘p’, ‘R’, and ‘l’
• Updated the description of Maturity Month, Day and Year
• Added the offset for ScreenLastTradeDate in New Expiry Message and New Option Market Definition Message

[Open-Markets-Initiative]

Thanks for taking a look at this. Our generator works on a "compiled" protocol where the most important source component is the Ice iMpact specification pdf. If you could post it here with a small pcap I could see what we can do about posting a new dissector.

[shatteringlass]

Thanks for the reply, your words really give me hope! Here's the ICE iMpact Multicast Feed Message Spec v1.1.33.1 and some PCAPs that could help.

[Open-Markets-Initiative]

Thanks for these. Provided there are no fundamental changes between the specification layouts I should have something by this weekend. I will keep you posted.

[Open-Markets-Initiative]

I posted first round of iMpact v1.1.34..I was able to get a copy of that specification. It turns out the sample captures look like version 1.1.33 (or some other earlier version) though many of the messages parse. Ice's policy on existing fields makes inline dissection annoying. Also, I will be adding some more detail to these (like enums etc).

[shatteringlass]

Great, I didn't notice a new spec was out. Thank you very much, I wish I could help with the points you made but I'm not sure I even understood.

[Open-Markets-Initiative]

I posted versions 1.1.33 and 1.1.34. Let me know if you find any issues. Also if you come across a packet with 4.1.21. SPECIAL FIELD MESSAGE (‘b’ ) please send it my way. This is a special case and some test data would be useful.

Lastly, if you have data for other protocols (pitch/opra) we are working on these blind.

[shatteringlass]

Actually I only just tested the dissector and it seems to fail due to error:

Lua Error: [string "/home/a462173/.local/lib/wireshark/plugins/Ic..."]:7231: Range is out of bounds

Which seems to refer to this line in the script:

local range = buffer(offset, length)

inside the Message dissector. Any chance you can look into this? Thanks

[Open-Markets-Initiative]

Can you check a couple of things:

1) The provided captures have been tested and verified with the v1.1.33 dissector...if you are dissecting a similar set of data, please make sure that only v1.1.33 dissector is in the plugins folder. Similarly if your data is v1.1.34 only that dissector should be in the plugins folder. Ice iMpact binary protocol fields are exact, and the dissectors will get off track if there are unexpected fields on the wire. Some protocols provide runtime 'on wire' identifying information, but as far as I know, iMpact lacks this, so we have to manually tell Wireshark which version to parse. You can also manually select the protocol in the menu with Analyze | Decode As .

2) If there is still an error, it means that there is an issue specific to that packet. Please provide a capture and I will update the compiled protocol and regenerate. You can find all problem packets with Analyze | Expert Information.

PS Individual packets can be exported with File | Export Specified Packets.

[shatteringlass]

You're right that I was trying to decode 1.1.33 pcaps with a 1.1.34 dissector, hence the error. After switching to the right dissector it seems to work fine. Thanks a lot.

[shatteringlass]

I have found the dissector to be broken against some files captured in the last week. I'm attaching one of those as an example. sample_pcap.zip

[Open-Markets-Initiative]

The invalid packets contain iMpact special fields. The special field message is complicated (almost a mini protocol in itself) and we needed test data. Sample data is crucial for debugging these special cases. Thanks for providing an example.

I posted a first round of regeneration that handles special fields by length, but we will upgrade the model in a week or so to handle the individual special fields using the type field. We have to upgrade the protocol compiler to handle a default path.