Field group got broken on message resend

[joaocpribeiro]

Version: 1.11.2

Steps:

• Using quickfixn on initiator side.
• Acceptor has network issues.
• Initiator sends a FIX message (NewOrderSingle), but it is not successful. -> QuickFix.Session.Send(...) returns false.
• Acceptor has their network working again.
• Logon is successful and acceptor sends a SequeceReset message.
• Acceptor sends the FIX message that failed previously.

Current behaviour:

• When the NewOrderSingle message is sent after SequenceReset, the order of the fields is changed from what was defined before. Field groups get broken.

Expected behaviour:

• When the NewOrderSingle message is sent after SequenceReset, the order of the fields is not changed. New fields (for example, 43-PossDupFlag) can be added in the adequate position. Field groups do not get broken.

Example data:

FIX message that was attempted to be sent while there were network issues: 8=FIXT.1.1|9=311|35=D|34=2278|49=EXAMPLE_INIT|52=20240812-14:38:39.223|56=EXAMPLE_ACCEPT|11=9560967b495d48cb8d3611cc2c61612d|15=EUR|21=1|22=4|38=5|40=2|44=50|48=DE0000000000|54=1|55=DE0000000000|59=6|60=20240812-14:38:38.913|100=MUNC|432=20250807|1133=G|10050=0000000|453=1|448=EXAMPLE_PARTY|447=D|452=24|802=1|523=K0000000000|803=22|10=146|

FIX message that was sent after SequeceReset: 8=FIXT.1.1|9=342|35=D|34=2278|43=Y|49=EXAMPLE_INIT|52=20240812-14:57:35.406|56=EXAMPLE_ACCEPT|122=20240812-14:38:39.223|11=9560967b495d48cb8d3611cc2c61612d|15=EUR|21=1|22=4|38=5|40=2|44=50|48=DE0000000000|54=1|55=DE0000000000|59=6|60=20240812-14:38:38.913|100=MUNC|432=20250807|447=D|448=EXAMPLE_PARTY|452=24|453=1|523=K0000000000|802=1|803=22|1133=G|10050=0000000|10=156

Please ignore the wrong CheckSum, I had to change some data from the messages. Please focus on tag 453 for NoPartyIDs group. This got broken.

Note that the messages do not have the field groups broken when the first sending attempt is successful.

[gbirchmeier]

When you first send any given message, the engine doesn't do any processing or verification on it. You created the message object, and the engine only calls ToString()* to get the string to send, and then it sends it. However, the message goes into the store as just as just a string.

*Note: a recent PR changes Message.ToString() to Message.ConstructString()

On the resend, the engine pulls that message string out of the store, and then re-constructs a Message object from it. NOW it uses a DD for construction. If the DD is configured correctly, no problem. If the DD is not, then the reconstruction is going to go badly, and that's what you're seeing.

So I believe this is occurring at Session.NextResendRequest(), where it calls

msg.FromString(msgStr, true, SessionDataDictionary, ApplicationDataDictionary, _msgFactory, ignoreBody: false);

Somehow for you, I think ApplicationDataDictionary is null. (Maybe SessionDataDictionary is too?)

So... what does your configuration look like?

[joaocpribeiro]

@gbirchmeier, my config looks like the following:

[DEFAULT]
ConnectionType=initiator
ReconnectInterval=5
FileStorePath={{FixFilesPath}}
FileLogPath=log
StartTime={{FixStartTime}}
EndTime={{FixEndTime}}
TimeZone={{FixTimeZone}}
UseDataDictionary=N
SocketConnectHost={{SocketConnectHost}}
SocketConnectPort={{SocketConnectPort}}
LogoutTimeout=5
ResetOnLogon=N
ResetOnDisconnect=N

[SESSION]
BeginString=FIXT.1.1
DefaultApplVerID=FIX.5.0
SenderCompID={{SenderCompID}}
TargetCompID={{TargetCompID}}
HeartBtInt=30

Some of the values are defined on runtime, but I believe that those have no impact on this issue. You referred Session.NextSendRequest(). Is this something I can catch on my code? Unfortunately this is a scenario that I cannot test easily. It was reported once after the Acceptor had issues on the connection. Since I had no answer in this thread for some time, I started preventing new NewOrderSingle messages when the session is off, but it might happen in the future that the session seems to be on and in the next step the Acceptor has issues already...

[gbirchmeier]

Ok, this is your problem:

UseDataDictionary=N

If you don't use a DD, then your app cannot correctly parse any message that has repeating groups in it, including those received from your counterparty. I don't know what state your project is in, but I'm surprised you haven't had problems well before this.

It's right there in the docs:

As I mentioned above, resends involve re-parsing the string message from the store. No DD? Then you can't correctly parse it.

Frankly, I don't know why we even allow UseDataDictionary=N. In my 15 years working with the various QF versions, I have never used it. There is no counterparty I have ever connected to that would work without a DD.

So:

• You need to customize your FIX50 DD, and possibly (but probably not) your FIXT11 DD, to match your counterparty's message structure. I'm 100% sure they've made customizations to messages or fields (because all counterparties do that), so your app needs to match.

• You need your config to include these:

  UseDataDictionary=Y
  TransportDataDictionary=path/to/your/custom/FIXT11.xml
  AppDataDictionary=path/to/your/custom/FIX50.xml

I'm going to close this issue, as it's clear to me now that there's not a QF bug here.

[gbirchmeier]

Oh, your other question:

You referred Session.NextSendRequest(). Is this something I can catch on my code?

No, this is internal engine code. I was just giving notes on my investigation. (And it's actually NextResendRequest(), I corrected that above)

[joaocpribeiro]

@gbirchmeier Thank you! We are still not in production, that's why we haven't had problems before. I started talks with my counterparty to check if they have something I can use to create my data dictionary. I will write again once we have it in place.