search

quickfix-j / quickfixj

QuickFIX/J is a full featured messaging engine for the FIX protocol. - This is the official project repository.
http://www.quickfixj.org
Other
964 stars 617 forks source link

SessionState#messageQueue is unbound #477

Closed philipwhiuk closed 10 months ago

philipwhiuk commented 2 years ago

Describe the bug The message queue in SessionState is unbound. This can lead to a vast quantity of messages being stored in memory which can lead to an OOM error.

In the worst case of a misbehaving FIX engine that doesn't handle resends, this queue grows without limit.

To Reproduce On the counter-party, induce the side to skip a large volume of sequence numbers. Then start sending messages at a high rate.

The QuickFIXJ FIX engine will cache all the received messages in memory while it waits for the counter-party to process the resend. This will use a potentially infinite amount of memory (but could easily store thousands of messages on a typical high volume system).

Expected behavior QFJ should only keep a (albeit size-able) limit of messages for a session. Any later messages should be dropped and then handled with a future re-send if necessary.

Implementation notes

Perhaps something like MaxMessagesQueuedWhilePendingResend=2000 We could allow -1 for people who are truly happy with unlimited.

chrjohn commented 2 years ago

Thanks @philipwhiuk , did you take a look at #380 or https://www.quickfixj.org/jira/browse/QFJ-968 respectively?

380 tries to tackle the same problem by adding the possibility to pass in a custom message queue.

philipwhiuk commented 2 years ago

Thanks @chrjohn. I didn't see QFJ-968 or the other PR. We implemented the PR I attached on our internal branch so I wanted to make it available to the community. But will look at both in more detail on Monday and follow-up with some more stuff.

chrjohn commented 10 months ago

Fixed via #652.