We notice that after running for a few days the Seq Forwarder service consumes 750mb+ of memory

[andwie]

After running for a few days the Seq Forwarder service consumes 750mb+ of memory. Is that normal? Is there a way to limit Seq Forwarder's memory usage similar to the Seq Server service? We do see entries in the seq-forwarder log files like the following:

2017-08-05 12:20:37.326 +00:00 [Warning] Buffer is full; dropping 1 events to make room for new ones 2017-08-05 12:33:43.396 +00:00 [Warning] Buffer is full; dropping 17 events to make room for new ones 2017-08-05 12:46:49.443 +00:00 [Warning] Buffer is full; dropping 10 events to make room for new one

Not sure if that is related.

[nblumhardt]

Hi Andrew,

It looks like the forwarder may be unable to send events on to Seq (or unable to send them on fast enough). Are you seeing events at the destination server?

Regards, Nick

[andwie]

Hi Nick, We reduced the number of events we are forwarding to the Seq Server. Now when I look at the logs for the Seq Forwarder I don't see that many dropped events:

2017-08-18 03:38:48.414 +00:00 [Warning] Buffer is full; dropping 1 events to make room for new ones 2017-08-18 07:38:00.893 +00:00 [Warning] Buffer is full; dropping 1 events to make room for new ones 2017-08-18 11:35:02.529 +00:00 [Warning] Buffer is full; dropping 2 events to make room for new ones

We still see high memory usage for the Seq Forwarder service. When I checked this morning it was consuming 1,048 MB.

[nblumhardt]

Thanks for the extra info Andrew.

Still doesn't sound quite like I'd expect.. Just a couple of things to check on the Seq side:

• Are you seeing the events from the forwarder in Seq?
• On the Seq server, under Settings > Diagnostics > Ingestion Log, do you see any entries?
• Are the payload size limits on Seq modified, i.e. what's the Raw event body limit under Settings > System?
• If the events are getting through fine, can you check their size by filtering down and running a query like: select length(@Document) from stream limit 5? (You might need to add a where clause to select just the relevant events)

Cheers!

[andwie]

Are you seeing the events from the forwarder in Seq? Yes

On the Seq server, under Settings > Diagnostics > Ingestion Log, do you see any entries? I just see this one entry: 2017-08-09 18:21:25 The ingestion API is configured and accepting requests

Are the payload size limits on Seq modified, i.e. what's the Raw event body limit under Settings > System? This setting is set to: 262144 bytes

If the events are getting through fine, can you check their size by filtering down and running a query like: select length(@Document) from stream limit 5? (You might need to add a where clause to select just the relevant events) I didn't see any large events: 654 659 662 1870 1887 1896 1868 1875 1859 1894 1871 1878 1862

I log about 60 events per minute currently. I'll make some additional reductions in the amount of events I log to the Seq Server and see if that reduces the memory footprint of the Seq Forwarder service.

[nblumhardt]

Interesting, thanks @andwie.

I am not as surprised about the memory usage (the storage is memory-mapped and sparse, so can show more memory allocated than is actually used), but the dropped events are still surprising. I don't think 60 events/minute are likely to stress things at all.

I'll crank this up again tomorrow and see if I can replicate the problem.

[nblumhardt]

Hi again,

I've been running 500 events/minute through steadily for a day, still no change in RAM usage, but I'll give it some more time:

I'm wondering if it's possible there's a configuration difference we're not thinking of? I'm running this on Win 10 with SSD (though not much I/O going on, at that ingestion rate).

Just wondering, does the RAM use slowly climb, or does it suddenly spike?

Thanks again for the info.

Nick

[Hinni]

Hi @andwie , @nblumhardt

We have the memory leak problems with some of our seq-forwarder instances, too. After a couple of days the memory footprint is over 900MB.

I think this only happens, if we have to patch our Seq server. In this minutes all event traffic is stored and the local data.mdb grow fast (we create many events on some servers). As soon as the seq server is back - all events will be sent without problems - but the local data.mdb don't shrink automaticly. It don't appears on instances with less traffic.

Instance A

Instance B

Maybe the Seq Forwarder has a problem with large local data.mdb? The CPU usage is high, too?! Or how can I provide you with more details?

Regards

[nblumhardt]

Hi @Hinni - thanks for sharing your data 👍

It would be good to try to identify whether you're seeing a leak, or just high memory consumption - does the process memory usage continue to grow indefinitely?

The data files may also be sparse - if you right-click and examine "size on disk", this may be substantially smaller than the reported file size. Any difference there?

Also, which OS are you running the forwarder on?

Many thanks, Nick

[Hinni]

Hi Nick,

The RAM usage is growing during a few days. I will check this on my development environment - where the limit is.

It seems to be that large

This instances are running on Windows Server 2012 R2 Standard Edition.

Regards

[Hinni]

In December and January I did not see this issue again. As soon as I got more details I will share it with you.

[Hinni]

I try the new version 1.1.80 to figure out if this large memory footprint is now better with commit #39. Our cloud test environment is offline over night. In the morning onprem forwarder has a big memory footprint. But I only would expect a large buffer data.mdb.

[nblumhardt]

All, I think what we're most likely to be seeing here is the result of LMDB's memory mapping of portions of the database, which expands the process's memory space, but doesn't generally use actual RAM.

Optimistically (and since it's not moving forward) I'll close this issue. In the long run, we'd like to move to a simpler storage layer for Forwarder, but we'll keep an eye on what we might do to improve in the meantime.