FieldNotFoundException - tag 271 not in message (35=x) where 269 = q

[derMaaster]

Hi, with data incremental refresh (35=x) the rule is that 271 is conditionally required when 279 = 0 AND 269 = 0, 1 or 2. I have incoming data from my market data provider where 269 = q (trading period details) and there is no 271 (MDEntrySize) field in the message then. This then is seen as an invalid message and I lose the fix connection.

Since 269 is not 0, 1 or 2 - that is then a bug correct?

This is with FixAntenna.NetCore

[derMaaster]

Is there a way I can change a setting so that the engine ignores 269 = q messages in the meantime?

[Viktar-Tserashchuk]

Hi @derMaaster, it seems just proper working validation. According to our dictionary, tag 269 cannot have value 'q'.

What you can do is either to turn off validation: https://github.com/epam/fix-antenna-net-core/blob/main/Docs/Configuration.md#validation

or update dictionary to allow value 'q'. The dictionaries can be found here: https://github.com/epam/fix-antenna-net-core/tree/main/FixAntenna/NetCore/Dictionaries FixAntenna tries to load dictionaries from the folder Dictionaries that is in the current working directory.

Just open the dictionary of the required FIX version, find <fielddef tag="269", update the list of the values accordingly, and put it in the folder Dictionaries where your application is.

When a FIX connection is made the engine will log something like: [DictionaryLoader]: Load resource:c:\projects\fa-net\Samples\EchoServer\bin\Release\net6.0\Dictionaries\fixdic44.xml This is the way you can check that the expected dictionary is loaded.

Here you can find more about the dictionary format: https://kb.b2bits.com/pages/viewpage.action?pageId=21364835

[derMaaster]

Hi @Viktar-Tserashchuk , I have validation=false in fixengine.properties. I will look at option 2 - to include q - this will then allow the engine to see that 269 is not equal to 0, 1 or 2 and thus not require 271 I presume. Thanks

[derMaaster]

Very easy fix it if it works! thanks

[derMaaster]

Hi @Viktar-Tserashchuk,

I still get the same invalid message error and disconnect with dictionary changed to:

`

Bid Offer Trade Index Value Opening Price Closing Price Settlement Price Trading Session High Price Trading Session Low Price Trading Session VWAP Price Auction Clearing Price Trading Period Details**

Type Market Data entry.

`

`

Conditionally required ifMDEntryType= Bid(0), Offer(1), or Trade(2)

`

fyi - I don't find '[DictionaryLoader]:' in my trace, info or debug logs on connect.

[derMaaster]

Unfortunately I can't test now as markets are closed already, can I adjust the rule to this:

`

Conditionally required ifMDEntryType= Bid(0), Offer(1), or Trade(2) andMDEntryType

`

and:

`

Conditionally required whenMDUpdateAction= New(0) andMDEntryType= Bid(0), Offer(1), or Trade(2) andMDEntryType= Auction Clearing Price(Q), Trading Period Details(q).

`

[Viktar-Tserashchuk]

Hi @derMaaster,

This is how it may look in the log when a dictionary is loaded from a file: The screenshot is a part of the log that our sample ConnectToGateway produces.

As for the condreq syntax, it looks okay to me. But the logic behind it seems a little bit odd. I would say that the first condition written that way would simply require existence of tag 271. It might be you want to remove the condreq attribute. That will make tag 271 optional.

You can find more examples of allowed expressions searching by condreq in the dictionary file.

[derMaaster]

I removed the condreq - that should be good then.

BUT, I do not see the DictionaryLoader in the debug log... I changed the fixdic42.xml file in the forked repo of FixAntenna Netcore which my app references. And the fixdic42.xml file in the bin folder of my app updates the changes correctly.

No reference of the dictionary loaded does concern met though

[Viktar-Tserashchuk]

Well, that probably means that the engine is not able to find the dictionary in the expected place and load the imbedded one. If this is the case then the log will also reflect it:

Antenna tries to find dictionaries in the following places:

1. ./Dictionaries
2. antenna dll's location/Dictionaries
3. a dictionary imbedded in the antenna's dll

[derMaaster]

Hi @Viktar-Tserashchuk,

I don't have anything of EmbeddedResourceLoader either:

[derMaaster]

I have made the changes to the fixdic42.xml in locations 1 and 2 you mention above. That also feeds through to my applicaitons fixdic42.xml that is at apps dll location/Dictionaries.

[OlehVasylievEPAM]

@derMaaster Hi, have you resolved your issue?

[derMaaster]

Hi @OlehVasylievEPAM,

No I have not. My service provider has possible values (for a 35=X message) for 269= q or Q.

I explain the details in the messages above - my opinion is that it is a bug with the Epam fix engine since tag 271 is only required if 269= 0, 1 or 2.. in any case. I have removed that conditional requirement in my dictionaries - but it seems not to load through and I still get that error and lose connectivity. Viktar has not responded to my last two messages.

[OlehVasylievEPAM]

@derMaaster

Could you share the problematic FIX message and your FIX engine config, please?

[derMaaster]

@OlehVasylievEPAM here it is:

8=FIX.4.2 | 9=156 | 35=X | 49=IRESSFIX6 | 56=RIDAPD | 34=72504 | 43=N | 52=20230911-08:23:06.863 | 262=Top | 268=1 | 279=0 | 55=YMAZ-NOV24-4740-P | 207=APD | 269=q | 272=20230911 | 273=08:23:06 | 336=COT | 10=009 |

` Log.TraceIsOn = true Log.Engine.TraceIsOn = true Log.DebugIsOn = true Log.Engine.DebugIsOn = true Log.NoteIsOn = true Log.WarnIsOn = true Log.ErrorIsOn = true Log.FatalIsOn = true

tradePeriodTimeZone = GMT+2 inMemoryQueue=true validation=false queueThresholdSize = 1

enableNagle=false

validateCheckSum=false validateGarbledMessage=false

markIncomingMessageTime=true

cpuAffinity=0`

[Viktar-Tserashchuk]

Hi @derMaaster,

I ran our EchoServer sample with your config parameters and sent the message to it several times. The sample was able to receive and send it back successfully. I also extracted tag 269 from the message in the sample and logged its value as part of the message handling.

public virtual void OnNewMessage(FixMessage message)
{
    Logger.Info("New message received: " + message);
    if (_echoMode)
    {
        Logger.Info($"Tag 269 = {message.GetTagValueAsString(269)}");
        _session.SendMessage(message);
    }
}

No dictionary modification involved.

It looks like FIxAntenna works okay and something else causes the issue. Also I don't think parameters like Log.NoteIsOn = true will work for FixAntenna .Net Core. They look like parameters from FixAntenna C++/.NET

[derMaaster]

Hi @Viktar-Tserashchuk ,

Thanks - I will do some troubleshooting with your echo-server and maybe with the FixClientSimulator

[derMaaster]

Hi @Viktar-Tserashchuk,

I got some time to do the same test with EchoSever and got error no Tag 270 (MDEntryPX) found, which is conditional if 279=0.

[2023-09-19 07:58:59.394] [ WARN] [MRThread<:testSession] [LoggingErrorHandler]: Invalid message received: MsgBuf{8=FIX.4.29=15235=X34=249=Rid56=EchoServer52=20230919-05:58:59.938262=Top268=1279=055=NOV23-3660-C207=APD269=q272=20230919273=05:58:59336=COT10=005} Invalid message - required tag missing. [SeqNum: 2, MsgType: X, Tag: 270]: 8=FIX.4.2 | 9=152 | 35=X | 34=2 | 49=Rid | 56=EchoServer | 52=20230919-05:58:59.938 | 262=Top | 268=1 | 279=0 | 55=NOV23-3660-C | 207=APD | 269=q | 272=20230919 | 273=05:58:59 | 336=COT | 10=005 |

When adding Tag 270 and removing Tag 207 (which it had a problem with). Then I get Tag 269 out of range. Then I update the Dictionary for extra possible value of q and Q:

`Auction Clearing Price

Trading Period Details` Then EchoServer abruptly shutsdown: `[2023-09-19 08:33:38.758] [ INFO] [MRThread<:testSession] [EchoServer]: New message received: 8=FIX.4.29=15435=X34=249=Rid56=EchoServer52=20230919-06:33:39.340262=Top268=1279=055=NOV23-3660-C269=q270=25000272=20230919273=06:33:39336=COT10=012 [2023-09-19 08:33:38.758] [DEBUG] [MRThread<:testSession] [AcceptorFixTransport]: >>8=FIX.4.2 | 9=154 | 35=X | 34=2 | 49=EchoServer | 56=Rid | 52=20230919-06:33:38.760 | 262=Top | 268=1 | 279=0 | 55=NOV23-3660-C | 269=q | 270=25000 | 272=20230919 | 273=06:33:39 | 336=COT | 10=017 | [2023-09-19 08:33:38.758] [ WARN] [MRThread<:testSession] [MessageReader]: Abrupt session testSession termination. ` The error and warn logs dont say much: `2023-09-19 09:02:15.5854 Epam.FixAntenna.NetCore.FixEngine.Session.LoggingErrorHandler Abrupt session testSession termination.System.IO.IOException: End of File read at Epam.FixAntenna.NetCore.FixEngine.Transport.FixMessageChopper.ReadAvailableBytesToBuffer() in C:\Users\allis\source\repos\Libraries\fix-antenna-net-core\FixAntenna\NetCore\FixEngine\Transport\FixMessageChopper.cs:line 674 at Epam.FixAntenna.NetCore.FixEngine.Transport.FixMessageChopper.ReadMessage(MsgBuf buf) in C:\Users\allis\source\repos\Libraries\fix-antenna-net-core\FixAntenna\NetCore\FixEngine\Transport\FixMessageChopper.cs:line 575 at Epam.FixAntenna.NetCore.FixEngine.Transport.AbstractFixTransport.ReadMessage(MsgBuf buf) in C:\Users\allis\source\repos\Libraries\fix-antenna-net-core\FixAntenna\NetCore\FixEngine\Transport\AbstractFixTransport.cs:line 106 at Epam.FixAntenna.NetCore.FixEngine.Session.IoThreads.MessageReader.Run() in C:\Users\allis\source\repos\Libraries\fix-antenna-net-core\FixAntenna\NetCore\FixEngine\Session\IoThreads\MessageReader.cs:line 216` `2023-09-19 09:02:15.5854 Epam.FixAntenna.NetCore.FixEngine.Session.IoThreads.MessageReader Abrupt session testSession termination.` Please give ideas what I can test next to troubleshoot?

[Viktar-Tserashchuk]

Hi @derMaaster,

By default, EchoServer's config has validation enabled. And this is different from what you have in your application. This is why you need all these manipulations with the dictionary. I would try testing with the exact same config parameters that you have for your production app.

Also, I did all the same changes to the dictionary and to the message you described and sent it. No problem.

I think that in your case the connection was closed by the other side. You can try to remove logs for both side and try again.

By the way, the exception says that the session was terminated, not the EchorServer. It still should be able to accept new connections.

[derMaaster]

Yes it is as you say. It is my application that ends the session with Tag 271 not found.

Please clarify what you mean by removing logs?

I have removed all the Log.<> settings in fixengine.properties

[Viktar-Tserashchuk]

Hi @derMaaster, I meant the folder with log files. I found it easier to start debugging with no previous history.

Yeah, the dictionaries are loaded when they are needed, and if validation is turned off then no dictionary is required. Sorry, if I misguided you in that aspect.

[derMaaster]

Hi @Viktar-Tserashchuk ,

No problem - When I turn validation=true / on. Then I get a warning but the connection is not broken - aka no error in the logs. When validation is off I always get error on that message.

Trying to update and change the dictionary while validation is on does not help. Adding possible values for 269. And making 271 not required for messages X and W.

[Viktar-Tserashchuk]

So, this is how I see it.

1. When validation is turned on, Antenna rejects messages that don't meet the dictionary's requirements. This looks correct.
2. When validation is turned off, Antenna can accept and send the problematic message back. The experiment with the EchoServer proved it. This part also looks correct.

It looks like your code that handle the message might cause the issue.

By the way, you wrote in you first message that "This then is seen as an invalid message and I lose the fix connection". Is there a log message about that or the connection just breaks?

[derMaaster]

Ok guess I wasted a lot of your time as well by not thinking rationally - the code breaks on:

int qty = fixMessage.GetTagAsInt(271);

Within OnNewMessage() , and because it breaks there, it "breaks" the fix engine.

[Viktar-Tserashchuk]

Hi @derMaaster,

Then I think you can just check the existence of the tag in the message end extract tag only if it exists. Something like

                if (fixMessage.IsTagExists(Tags.MDEntrySize))
                {
                    int qty = fixMessage.GetTagAsInt(Tags.MDEntrySize);
                    Logger.Info($"Tag {Tags.MDEntrySize} = {qty}");
                }
                else
                {
                    Logger.Info($"Tag {Tags.MDEntrySize} = Not Found");
                }

[derMaaster]

Thank you @Viktar-Tserashchuk ,

I think that concludes this issue. I appreciate the help