Huge memory usage

[sachabarber]

Overview

So we use QuickFixN within some production code. We have observed that over a period of 4-5 days the memory footprint of our app grows up to around 2GB.

We profiled the app, and can see that the cause appears to be FileStore where the offsets_ Dictionary is storing too much data

One thing to note, is that left alone, our app rarely restarts

I have downloaded the QuickFixN code and been digging about to see where the offsets_ is used and how it is cleared, and the following is what I see

FileStore

The FileStore class contains the offsets_ which is only cleared, when the ConstructFromFileCache() method is called. Which in turn is called when the FileStore.open() method is called, which is only called under 2 conditions

• FileStore construction
• Reset() called on FileStore

So I was thinking perhaps the Reset needs to be called. But by whom and when?

SessionState

So I dug some more, and can see that the FileStore.Reset() is called from SessionState.Reset(string reason). Which in turn is called by various other call sites from Session.

Session

• The calls to SessionState.Reset() looks like an integral part of the QuickFixN implementation, so these call sites are obviously correct and tested by this repo. Basically I trust the existing Session calls to to SessionState.Reset()

So the question

Is this memory footprint normal, or is specific QuickFixN application code expected to also call the Session.Reset() which will call the callsites I mention above? Where the end result could be a call to the FileStore.ConstructFromFileCache.

However looking into this, I am not sure a Reset would work, as from what I can tell the FileStore.ConstructFromFileCache() method would read the contents of the files shown in image below to construct the offsets_. Which is obviously will just fill the offsets_ straight back up.

private void ConstructFromFileCache()
{
    offsets_.Clear();
    if (System.IO.File.Exists(headerFileName_))
    {
        using (System.IO.StreamReader reader = new System.IO.StreamReader(headerFileName_))
        {
            string line;
            while ((line = reader.ReadLine()) != null)
            {
                string[] headerParts = line.Split(',');
                if (headerParts.Length == 3)
                {
                    offsets_[Convert.ToInt32(headerParts[0])] = new MsgDef(
                        Convert.ToInt64(headerParts[1]), Convert.ToInt32(headerParts[2]));
                }
            }
        }
    }

    if (System.IO.File.Exists(seqNumsFileName_))
    {
        using (System.IO.StreamReader seqNumReader = new System.IO.StreamReader(seqNumsFileName_))
        {
            string[] parts = seqNumReader.ReadToEnd().Split(':');
            if (parts.Length == 2)
            {
                cache_.SetNextSenderMsgSeqNum(Convert.ToInt32(parts[0]));
                cache_.SetNextTargetMsgSeqNum(Convert.ToInt32(parts[1]));
            }
        }
    }
}

If I look at the current memory usage of our PROD QuickFixN code, which is using these files, the memory footprint is pretty much exactly the total bytes for these files, which is what I would expect looking at how the FileStore.ConstructFromFileCache() method works (shown above)

So not really sure that the Reset approach is going to help that much

So what choices do we have to maintain a decent memory footprint for our application?

Could someone advice what we should be looking at doing?

Right now we restart the app manually every 5 days, and from there it builds up to about 2GB over 5 days, but we would be keen to find a more programmatic solution

[tomerpeled]

Any solution for that?

[gbirchmeier]

The store is needed to support the ResendRequest functionality. This is the cache of messages that will be resent if a ResendRequest situation occurs.

Calling Reset() in user-code is a not an appropriate response; you're wiping out the store, and thus eliminating the ability to resend. Internally to QF/n, I believe Reset() is only called when the session restarts.

However, I agree that 2GB is over the top, and the scenario where we'd need to replay a whole day or more is unlikely. I'm just thinking in text right now, but perhaps we could have a configurable upper-store limit, like a max number of messages or a max number of hours/minutes ago. (Actually we do have a MaxMessagesInResendRequest, but I don't think the any store impls are looking at it.)

The key problem here is that the store implementations aren't looking at any other settings than FileStorePath. For instance, If Resend is disabled, then the store doesn't need to store any messages content. If the store size is limited by MaxMessagesInResendRequest, then the store could be able to trash the oldest stuff.

I'm curious to see if QF/j handles this differently.

I'll keep this issue in my mental queue, and come back to it after I knock a few other things out.