URGENT: Message Field Ordering

pakheili commented 8 months ago

Hi @TimelordUK. Great work on this library, thank you very much.

I hope this message finds you well. We have been utilising the library for managing FIX messages and have encountered a specific issue regarding the ordering of fields (tags) within the message body. Currently, the library appears to sort the fields in a standard order, which does not have the flexibility to handle unsorted fields in a message.

The problem arises when our clients send market data request messages, where the order of the fields in the message body is as follows: 49,56,34,52,262,263,264,146,461,55 (the positions of tags 461 and 55 are swapped). We reject this order because it does not follow the standard order.

However, the library accepts the order of the fields as 49,56,34,52,262,263,264,146,55,461.

Below is the standard ordered message that the client sends us and they successfully receive market data:

8 (BeginString) =FIX.4.4
9 (BodyLength) =0000115
35 (MsgType) =V
49 (SenderCompID) =sender
56 (TargetCompID) =target
34 (MsgSeqNum) =3
52 (SendingTime) =20231108-09:40:46.257
262 (MDReqID) =1698937860913.38
263 (SubscriptionRequestType) =1
264 (MarketDepth) =85
146 (NoRelatedSym) =1
55 (Symbol) =*
461 (CFICode) =O
10 (CheckSum) =033

To address this issue, we kindly request your guidance on the following matters:

Automatic Sorting: Is there a built-in mechanism within the library that can be utilised to automatically sort the various fields within the three components (header, body, trailer) of messages by tag number before receiving a request message in a session? If so, could you provide us with documentation or guidance on how to achieve this behaviour?
Field Ordering Customisation: Can we customise the order of the fields in a message sent to our acceptor using the library? If such customisation is possible, we would appreciate any insights or instructions on how to achieve this flexibility.
Preventing Sorting Behaviour: In case the sorting behaviour is a default behaviour of the library, is there a way to prevent the automatic sorting of fields, ensuring that the order of fields in the message body remains unchanged? We would appreciate any guidance on this matter.

We value the functionality and reliability of the library and believe that addressing this issue or obtaining the requested information would greatly enhance our integration with the library.

Thank you for your assistance in advance. We look forward to your response and any guidance you can provide to help us resolve this matter.

TimelordUK commented 8 months ago

What error do we reject with out of interest. What is fix specification file you are using. In theory we should not reject based on field order I’ll have to first reproduce this problem. We make two passes over a meaaage. First we simply mark where each field starts and end and we parse the tag. Then we send it through a segment parser where each tag is checked and we add structure. For example if a field belongs in instrument we start an instrument segment. We sort the message fields at end when for example you ask for a certain field it can be searched with binary search. All structure is known even after this so we know where everything was. Can you send exact message transactions you can remove the header for confidentiality if you wish I’d like to see exactly what error is sent. Sent from my iPhoneOn 8 Nov 2023, at 11:29, Pak Hei Li @.**> wrote: Hi @TimelordUK. Great work on this library, thank you very much. I hope this message finds you well. We have been utilising the library for managing FIX messages and have encountered a specific issue regarding the ordering of fields (tags) within the message body. Currently, the library appears to sort the fields in a standard order, which does not have the flexibility to handle unsorted fields in a message. The problem arises when our clients send market data request messages, where the order of the fields in the message body is as follows: 49,56,34,52,262,263,264,146,461,55 (the positions of tags 461 and 55 are swapped). We reject this order because it does not follow the standard order. However, the library accepts the order of the fields as 49,56,34,52,262,263,264,146,55,461. Below is the standard ordered message that the client sends us and they successfully receive market data: 8 (BeginString) =FIX.4.4 9 (BodyLength) =0000115 35 (MsgType) =V 49 (SenderCompID) =sender 56 (TargetCompID) =target 34 (MsgSeqNum) =3 52 (SendingTime) =20231108-09:40:46.257 262 (MDReqID) =1698937860913.38 263 (SubscriptionRequestType) =1 264 (MarketDepth) =85 146 (NoRelatedSym) =1 55 (Symbol) = 461 (CFICode) =O 10 (CheckSum) =033

To address this issue, we kindly request your guidance on the following matters:

Automatic Sorting: Is there a built-in mechanism within the library that can be utilised to automatically sort the various fields within the three components (header, body, trailer) of messages by tag number before receiving a request message in a session? If so, could you provide us with documentation or guidance on how to achieve this behaviour?

Field Ordering Customisation: Can we customise the order of the fields in a message sent to our acceptor using the library? If such customisation is possible, we would appreciate any insights or instructions on how to achieve this flexibility.

Preventing Sorting Behaviour: In case the sorting behaviour is a default behaviour of the library, is there a way to prevent the automatic sorting of fields, ensuring that the order of fields in the message body remains unchanged? We would appreciate any guidance on this matter.

We value the functionality and reliability of the library and believe that addressing this issue or obtaining the requested information would greatly enhance our integration with the library. Thank you for your assistance in advance. We look forward to your response and any guidance you can provide to help us resolve this matter.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

TimelordUK commented 8 months ago

I should add when sending a message it should be formatted exactly as declared in specification. The engine will go through each field one at a time and if present will send that field at that position. If you for example change the ordering in the xml file the engine from acceptor to client should send the message in that order. Sent from my iPhoneOn 8 Nov 2023, at 18:54, Stephen James @.> wrote:What error do we reject with out of interest. What is fix specification file you are using. In theory we should not reject based on field order I’ll have to first reproduce this problem. We make two passes over a meaaage. First we simply mark where each field starts and end and we parse the tag. Then we send it through a segment parser where each tag is checked and we add structure. For example if a field belongs in instrument we start an instrument segment. We sort the message fields at end when for example you ask for a certain field it can be searched with binary search. All structure is known even after this so we know where everything was. Can you send exact message transactions you can remove the header for confidentiality if you wish I’d like to see exactly what error is sent. Sent from my iPhoneOn 8 Nov 2023, at 11:29, Pak Hei Li @.> wrote: Hi @TimelordUK. Great work on this library, thank you very much. I hope this message finds you well. We have been utilising the library for managing FIX messages and have encountered a specific issue regarding the ordering of fields (tags) within the message body. Currently, the library appears to sort the fields in a standard order, which does not have the flexibility to handle unsorted fields in a message. The problem arises when our clients send market data request messages, where the order of the fields in the message body is as follows: 49,56,34,52,262,263,264,146,461,55 (the positions of tags 461 and 55 are swapped). We reject this order because it does not follow the standard order. However, the library accepts the order of the fields as 49,56,34,52,262,263,264,146,55,461. Below is the standard ordered message that the client sends us and they successfully receive market data: 8 (BeginString) =FIX.4.4 9 (BodyLength) =0000115 35 (MsgType) =V 49 (SenderCompID) =sender 56 (TargetCompID) =target 34 (MsgSeqNum) =3 52 (SendingTime) =20231108-09:40:46.257 262 (MDReqID) =1698937860913.38 263 (SubscriptionRequestType) =1 264 (MarketDepth) =85 146 (NoRelatedSym) =1 55 (Symbol) =* 461 (CFICode) =O 10 (CheckSum) =033

To address this issue, we kindly request your guidance on the following matters:

Automatic Sorting: Is there a built-in mechanism within the library that can be utilised to automatically sort the various fields within the three components (header, body, trailer) of messages by tag number before receiving a request message in a session? If so, could you provide us with documentation or guidance on how to achieve this behaviour?

Field Ordering Customisation: Can we customise the order of the fields in a message sent to our acceptor using the library? If such customisation is possible, we would appreciate any insights or instructions on how to achieve this flexibility.

Preventing Sorting Behaviour: In case the sorting behaviour is a default behaviour of the library, is there a way to prevent the automatic sorting of fields, ensuring that the order of fields in the message body remains unchanged? We would appreciate any guidance on this matter.

We value the functionality and reliability of the library and believe that addressing this issue or obtaining the requested information would greatly enhance our integration with the library. Thank you for your assistance in advance. We look forward to your response and any guidance you can provide to help us resolve this matter.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

TimelordUK commented 8 months ago

so i have written a test here on fix 52 branch

note ordering as 461=0|55=GBPUSD|

we still seem to parse the message and see the instrument view. So im not sure the reason for the reject is down to rerversed fields - it is possible the current rellease does behave this way

test('parse MD request with instrument 461=0||55=GBPUSD|', async () => {
  // const changed = logon.replace('108=62441|','000=62441|')
  const msg = '8=FIX.5.0SP2|9=0000145|35=V|49=init-comp|56=accept-comp|34=2|57=fix|52=20231108-20:13:54.831|262=#GBPUSD#0#|263=1|264=0|267=3|269=0|269=1|269=H|146=1|461=0|55=GBPUSD|10=109|'
  const res = await setup.client.parseText(msg)
  expect(res.view).toBeTruthy()
  const o = res.view?.toObject() as IMarketDataRequest
  const grp = o.InstrmtMDReqGrp?.NoRelatedSym ?? []
  expect(grp.length).toEqual(1)
  console.log(JSON.stringify(o, null, 4))
  expect(grp[0].Instrument).toEqual({
    Symbol: 'GBPUSD',
    CFICode: '0'
  })
})

pakheili commented 7 months ago

Hi @TimelordUK, thank you for your help and detailed response!

Here are some clarifications, and hopefully it will let you better understand our issue:

We are using FIX.4.4 as the convention. These are the Market Data Request messages that were sent by our client. I don't remember which one was it in the logs, but one of our client's request messages returns an error because the |461=O|55=*| and |55=*|461=O| after tag 146 (NoRelatedSym) are swapped in positions, which results in our server returning an unknown tag type 146 error message. Hence, I believe we rejected our client's message order because it does not follow the standard conventional order.

Note that tag 461 (CFICode) is the alphabet 'O', not the number '0'. We are using the current stable release jspurefix 3.0.0.

If there are any issues with this existing release, please let us know the best solution to patch the package.

Market Data Request Message with after tag 146 - `|461=O|55=|` = `|CFICode=O|Symbol=|`

8=FIX.4.4|9=0000115|35=V|49=init-comp|56=accept-comp|34=2|57=fix|52=20231108-09:40:46.257|262=1698937860913.38|263=1|264=0|267=3|269=0|269=1|269=H|146=1|461=O|55=*|10=033|

Market Data Request Message with after tag 146 - `|55=|461=O|` = `|Symbol=|CFICode=O|`

8=FIX.4.4|9=0000115|35=V|49=init-comp|56=accept-comp|34=2|57=fix|52=20231108-09:40:46.257|262=1698937860913.38|263=1|264=0|267=3|269=0|269=1|269=H|146=1|55=*|461=O|10=033|

Server's Error Messages Upon Receiving Our Client's Market Data Request Message

info - {
    "timestamp": "2023-11-14T13:07:28.336Z",
    "level": "ERROR",
    "app_name": "FIXSERVER-POOL2-SPECTRAOPTIONCUTTEST3",
    "service": "FIXSERVER-POOL2-SPECTRAOPTIONCUTTEST3:FIXSESSION",
    "message": "unknown tag type 146",
    "data": {
        "error": "at Writable.write (node:internal/streams/writable:337:10)"
    }
}

info - {
    "timestamp": "2023-11-14T13:07:28.336Z",
    "level": "WARNING",
    "app_name": "FIXSERVER-POOL2-SPECTRAOPTIONCUTTEST3",
    "service": "FIXSERVER-POOL2-SPECTRAOPTIONCUTTEST3:FIXSESSION",
    "message": "rx error event: unknown tag type 146 Error: unknown tag type 146\n    at examine (/usr/fix-engine/node_modules/jspurefix/src/buffer/ascii/ascii-segment-parser.ts:88:17)\n    at discover (/usr/fix-engine/node_modules/jspurefix/src/buffer/ascii/ascii-segment-parser.ts:130:27)\n    at AsciiSegmentParser.parse (/usr/fix-engine/node_modules/jspurefix/src/buffer/ascii/ascii-segment-parser.ts:156:5)\n    at AsciiParser.getView (/usr/fix-engine/node_modules/jspurefix/src/buffer/ascii/ascii-parser.ts:183:62)\n    at AsciiParser.msg (/usr/fix-engine/node_modules/jspurefix/src/buffer/ascii/ascii-parser.ts:85:42)\n    at AsciiParser.parse (/usr/fix-engine/node_modules/jspurefix/src/buffer/ascii/ascii-parser.ts:168:14)\n    at Writable.write [as _write] (/usr/fix-engine/node_modules/jspurefix/src/buffer/ascii/ascii-parser.ts:56:16)\n    at writeOrBuffer (node:internal/streams/writable:392:12)\n    at _write (node:internal/streams/writable:333:10)\n    at Writable.write (node:internal/streams/writable:337:10)"
}

Functional Internal Test Client Market Data Request That Meets Our Server's Requirements

    public createOptionCutMarketDataRequest(): Partial<IMarketDataRequest> {
        return {
            MDReqID: `#OPTION-CUT#`,
            SubscriptionRequestType: SubscriptionRequestType.SnapshotPlusUpdates,
            MarketDepth: 1,
            InstrmtMDReqGrp: {
                NoRelatedSym: [
                    {
                        Instrument: {
                            CFICode: 'O',
                        },
                        UndInstrmtGrp: {},
                        InstrmtLegGrp: {},
                    },
                ],
            },
        };
    }

TimelordUK commented 7 months ago

What XML file are you using on server for the data dictionary

TimelordUK commented 7 months ago

the problem i have currently is I can run that message through the 3.0.0 fix engine using FIX44.xml and it seems to work with these fields swapped.

const marketDataMsg = '8=FIX.4.4|9=0000170|35=V|49=init-comp|56=accept-comp|34=2|57=fix|52=20231108-09:40:46.257|262=1698937860913.38|263=1|264=0|267=3|269=0|269=1|269=H|146=1|461=O|55=*|10=033|'

 {
        "StandardHeader": {
            "BeginString": "FIX.4.4",
            "BodyLength": 170,
            "MsgType": "V",
            "SenderCompID": "init-comp",
            "TargetCompID": "accept-comp",
            "MsgSeqNum": 2,
            "TargetSubID": "fix",
            "SendingTime": "2023-11-08T09:40:46.257Z"
        },
        "MDReqID": "1698937860913.38",
        "SubscriptionRequestType": "1",
        "MarketDepth": 0,
        "MDReqGrp": {
            "NoMDEntryTypes": [
                {
                    "MDEntryType": "0"
                },
                {
                    "MDEntryType": "1"
                },
                {
                    "MDEntryType": "H"
                }
            ]
        },
        "InstrmtMDReqGrp": {
            "NoRelatedSym": [
                {
                    "Instrument": {
                        "Symbol": "*",
                        "CFICode": "O"
                    }
                }
            ]
        },
        "StandardTrailer": {
            "CheckSum": "033"
        }
    }

can you try and run a test client where for that client (leave server exactly as it is) you change the FIX44.xml file around line 2430 switch the two fields in question. Then run your test sending a test message to the server and your test should then send like your client is sending with fields reversed. Maybe safer to copy the FIX44.xml file change it as below and use the new file name in the initiatior.json config

{
  "application": {
    "type": "initiator",
    "name": "test_client",
    "tcp": {
      "host" : "localhost",
      "port": 2344
    },
    "protocol": "ascii",
    "dictionary": "data/FIX44.xml"
  },
  "Username": "js",
  "Password": "pwd",
  "EncryptMethod": 0,
  "ResetSeqNumFlag": true,
  "HeartBtInt": 30,
  "SenderCompId": "init-comp",
  "TargetCompID": "accept-comp",
  "TargetSubID": "fix",
  "BeginString": "FIX.4.4"
}

Does that then cause this exception.

This doesnt look like an easy fix we have to understand exactly what is going on.

if you can reproduce the problem that would be a start. Clearly I see you are seeing this exception in the log the trouble is I cannot reproduce it.

 <component name='Instrument'>
   <field name='Symbol' required='N' />
   <field name='SymbolSfx' required='N' />
   <field name='SecurityID' required='N' />
   <field name='SecurityIDSource' required='N' />
   <component name='SecAltIDGrp' required='N' />
   <field name='Product' required='N' />
   <field name='CFICode' required='N' />

TimelordUK commented 7 months ago

out of interest how many concurrent sessions would be running on same node instance i.e. how do you fan out the connections is it for example the case the node instance is under load. Can problem be reproduced with a server with no active connections and the test sent as above.

TimelordUK / jspurefix