Handling incoming messages while wating for Terminate response

[ujos]

Should Terminate Initiator handle Retransmit Request, business level messages and other incoming messages while waiting for Terminate Ack?

[donmendelson]

I assume that your question is about a recoverable flow in the absence of logical flow finalization (FinishedSending/FinishedReceiving).

Yes, business messages on any flow type should be processed until Terminate acknowledgement is received. That is purpose of waiting for the acknowledgement--to make sure that in-flight messages are processed before closing the connection.

The spec is currently silent on whether RetransmitRequest should be handled in this case. This should be addressed. (It is explicit that retransmission is allowed before finalization is complete.)

[ujos]

Yes, business messages on any flow type should be processed until Terminate acknowledgement is received.

So, it means that if Server sends Terminate message and receives an Order from client, then Server should send Execution Report back after Terminate message is sent...

[kleihan]

The server sends the Terminate message for a reason, i.e. server should respond until the Terminate ack comes in, but not necessarily with an ExecutionReport. Server can just as well send a BusinessMessageReject to say that the application is no longer available to take in (and respond to) transactions.

[adkapur]

If the terminate reason is ungraceful due to an error condition then the transport will be terminated period

If it is a graceful termination without finished sending then retransmission request will be serviced from counterparty but no business message so what hanno says will apply (orders will be rejected)

Same is the case for server sending FinishedSending i.e. if orders are received from counterparty after that then those will be rejected since only retransmission requests will be serviced from that point onwards

[ujos]

Same is the case for server sending FinishedSending i.e. if orders are received from counterparty after that then those will be rejected since only retransmission requests will be serviced from that point onwards

If server has Finished flow, it cannot send any message back. Even EexcutionReport(Rejected).

[ujos]

However... what if client has sent few orders at the same time when server sent Finished Sending. So, looks like Server should be able to send business level messages after FInishedSending message. But new issued Reject message will consume sequence number, and FInishedSending has already fixed last sent Sequence Number and it cannot increment it...

[ujos]

@donmendelson , There is still unclear case: how to react on incoming message after FinishedSending is sent.

[donmendelson]

@ujos I assume that you mean how should the sender of FinishedSending react to incoming application messages. The spec can be more explicit.

I think the answer is that it continues to process application messages until FinishedReceiving is received. Any subsequent application messages would be a protocol violation, but the session will be terminated in either case. The sender of FinishedSending may not send any application message after that event. To suggest otherwise could result in an infinite loop and prevent the session from every closing. The only messages it may send then are Retransmission if requested and FinishedSending at intervals as heartbeats.

[ujos]

I assume that you mean how should the sender of FinishedSending react to incoming application messages. The spec can be more explicit.

Yes, I do.

To suggest otherwise could result in an infinite loop

Why does it result infinite loop?

The sender of FinishedSending may not send any application message after that event.

So, it is ok to send application level messages after FinishedSending message. Correct?

[donmendelson]

Why does it result infinite loop?

It shouldn't if we follow the rules--no application message sent after a party sends FinishedSending or FinishedReceiving. I was trying to say if we make exceptions, then the session may never get finalized.

So, it is ok to send application level messages after FinishedSending message. Correct?

No, I did not mean to imply that.

[ujos]

No, I did not mean to imply that.

What if someone sends order and order is filled? Manual intervention is required to figure that out.

[donmendelson]

Sending FinishedSending means that the party has no intention of conducting business on this session ever again. If there is still a possibility that orders will be filled, then the session should not be finalized.

[kleihan]

The sender must be aware of the fact that he has declared the end of application messages by means of FinishedSending. Hence it is an error if he would continue to send them as it represents conflicting information towards the counterparty. Such behaviour should not be supported by the protocol.

[ujos]

Right... But what bothers me, is that Trader can send order and nothing will be sent back.

Looks like before sending FinishSending exchange should switch to pre-closing state and send notification to Traders. It is a signal for Trading systems to stop their order sending loop. In this state Exchange will reject all incoming orders.

Then, after few minutes, Exchange will move to closed state by issuing FinishSending.

[kleihan]

@ujos: The solution is that the trader should not be able to do so. It is important to distinguish session and application layer. When I said "the sender must be aware", I referred to the application being the one deciding on a business level that no more orders etc. need to be sent. That would be the trigger for the session layer to send FinishedSending. Your architecture seems to be different, i.e. that the session layer sends FinishedSending "on its own", e.g. time triggered. It should be the application that tells the session that it will no longer pass application messages for transmission. That should trigger the session layer to use FinishedSending to tell the counterparty that he does not have to expect further application messages.

[ujos]

Looks like we are talking about different use cases:

• Issuing FinishSending by Trader
• Issuing FinishSending by Exchange

In case of Trader, stop procedure is simple and works like you say:

1. Issue FinishSending
2. Wait for FinishedReceiving. Apply incoming Execution Reports but do not create new Orders.
3. Exit Orders Sending Loop.

In case of Exchange it is a bit more complicated. Exchange is not an Initiator of the Order. It is an Acceptor. Exchange cannot force others to not send Orders. It should react somehow on all incoming messages. From other side, after Trader has sent and Order, he/she needs to know either it is accepted or not.

So, returning back to my question, after Exchange has send FinishedSending message, it cannot reject incoming order, because Reject message consumes sequence number.

[kleihan]

It looks like this is part of your application architecture/design where you chose the application to have two triggers for FinishedSending on the same session. You will probably have good reasons for that but it is hard to comment upon. Generally speaking, the one or more applications using the same session need to synchronize in terms of who and when a FinishedSending is sent to the counterparty. When you say "Exchange cannot force others to not send Orders" it sounds like two independent applications/components are using the same session. In your terminology, "Trader" and "Exchange" need to work together to figure out the point in time when they can (jointly) close down the session. My apologies if I have not understood your use cases properly and am on the wrong track here.

[adkapur]

In theory we will close the market, send eliminations followed by FinishedSending and end of story

In reality we are planning to do this only for end of the week since it is difficult for gateway to know about match engine trading state

After FinishedSending is sent and then application message is received from counterparty then it will be rejected using Terminate since it is a protocol violation and not execution report reject or business reject and Terminate message does not consume a sequence number

The only message counterparty can send at that point (if it does not reply back with FinishedReceiving) is RetransmissionRequest

[ujos]

After FinishedSending is sent and then application message is received from counterparty then it will be rejected using Terminate since it is a protocol violation and not execution report reject or business reject and Terminate message does not consume a sequence number

Why is it protocol violation?

[kleihan]

Section 7.4.5 of the FIXP specification should be the right place to clarify the given use case. Currently, it does not show what happens if the receiver of a FinishedSending message sends further application messages and how to respond to them. FIXP RC4 is open for public comment at https://forum.fixtrading.org/t/public-comment-period-fix-performance-session-layer-fixp-release-candidate-4/14033 until end of July.

[ujos]

@kleihan, it is impossible to verify, either remote side has received FinishedSending message or not. Transport is asynchronous. Thus, verification of the incoming message type locally does not make sense.

[kleihan]

My understanding is that FinishedReceiving is the response confirming that FinishedSending has been received. Please post a public review comment under the link above with a suggestion on how the FIXP spec or the scenarios in section 7.4 should be changed. You can register for free as a user on the FIX website. The audience is much larger there than here and it is intended to be the place to capture feedback on release candidates.

[adkapur]

Please see section 7.4.6 thanks

Finished Sending & Half-Close Once one of the two parties has ceased logical flow of messages from its connection at the end of the day, end of the week or upon market close then it should still be ready and able to accept messages from the other counterparty till the time that the counterparty itself does not cease logical flow of messages from its own connection. However this should not lead to any corresponding output back from the connection which has been half-closed (with the exception of Retransmission) since that would be a protocol violation and lead to ungraceful termination.

It is a protocol violation since I cannot acknowledge or even reject an order once it is sent after a FinishedSending has been sent and the only response will be a Terminate

The only responses I am allowed to send after sending a FinishedSending message is to service a Retransmission request

Let's not split hairs about in-flight messages crossing each other over the TCP connection since this is still an improvement from today when an order could be sent in-flight while the market data security status message is published indicating market close/halt etc

[ujos]

It is a protocol violation since I cannot acknowledge or even reject an order once it is sent after a FinishedSending has been sent and the only response will be a Terminate

I believe termination is not good solution. If it is violation, how can remote counterparty avoid termination?

I think the best, what can be done in this case, is to send some Session Level Notification to remote side, that message is not delivered to the application layer and will be ignored.

[adkapur]

In that case the only option is a NotApplied message but it is considered to be an application message which consumes a sequence number

Manybe send another FinishedSending after NotApplied with the updated LastSeqNo then

[ujos]

Manybe send another FinishedSending after NotApplied with the updated LastSeqNo then

That would be great.

[kleihan]

The intention of FinishedSending is to allow the recipient to take action based on the understanding that there will be no more application related information after that. Having to assume that there is at least one case where there is a second FinishedSending does not allow such action. There could be any number of such messages and one would never be sure whether it was the last one. I do not see the benefit of such a workflow. Terminating a session due to unexpected (non-compliant) behaviour by the counterparty is a valid approach as it most likely requires manual intervention to overcome this kind of error.

[ujos]

@kleihan , if one sends message and he cannot be sure, either message is delivered, this is non-reliable protocol. However, according to requirements, protocol must be reliable:

[02Requirements.md] Create an enhanced session protocol that can provide reliable, highly efficient, exchange of messages to support high performance financial messaging, over a variety of transports.

Remote side should either acknowledge or reject incoming message.

[kleihan]

Reliable also means that there is a defined behaviour. The termination of a session reliably means that all of your previously sent and not explicitly acknowledged/rejected messages have been rejected. The termination is your response. There is no requirement to process every application message by means of the application. The session layer may also deal with it and respond to it.

[adkapur]

but this is an in-flight situation and comparing timestamp of the order sent with the timestamp of FinishedSending would have helped but both Terminate and FinishedSending lack a timestamp

[kleihan]

Timestamps only help to establish a sequence of events if they are highly synchronized. I would still say that all of previously sent and not explicitly acknowledged/rejected messages have not been processed, i.e. nothing changed on the other side with regards to what is intended by the payload.

[adkapur]

Yes agreed but still something is better than nothing and ideally both sides are using PTP or something

Generally speaking silent failures of dropped orders are anathema and lead to more confusion with respect to things like were they counted in pre-trade risk checks or not etc

[donmendelson]

One clarification suggested by the XMIT spec: If the connection drops after FinishedSending but before FinishedReceiving, then finalization has failed. In that case, the client may reestablish the session in order to reattempt graceful finalization.