Exception of type 'System.OutOfMemoryException' was thrown.

[bdw429s]

Just upgraded to Boncode version 1.0.41

I have a client whose site became unresponsive for a while, returning the "Generic Connector Communication Error" for a while. Max connections is not set in the settings file (defaulting to 0 then) and I can confirm that Lucee/Tomcat were not restarted. In fact, there is nothing at all in any of the Lucee, Tomcat, or Catalina log files that happened at that time.

However, several errors such as the following did appear in the Boncode logs, which are set to log level 3 currenlty.

2018-12-06 19:05:43 New Connection 21246 of 0 to tomcat: [::1]:8009 ID: 21243 [T-862]
2018-12-06 19:05:48 1.0.41 ERROR 
TCP Client level -- Server/Port:localhost/8009
Exception of type 'System.OutOfMemoryException' was thrown.
   at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat()
   at BonCodeAJP13.BonCodeAJP13ServerConnection.HandleConnection()
   at BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend)
2018-12-06 19:05:48 New Connection 21248 of 0 to tomcat: [::1]:8009 ID: 21244 [T-849]

2018-12-06 19:08:30 1.0.41 ERROR 
Stream reading problem (2)(2), we stopped waiting on Tomcat response. You may have shutdown Tomcat unexpectedly
Exception of type 'System.OutOfMemoryException' was thrown.
   at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat()
2018-12-06 19:08:30 failed Server Connection was aborted

2018-12-06 19:08:30 1.0.41 ERROR 
TCP Client level -- Server/Port:localhost/8009
Connection between Tomcat and IIS experienced error. If you restarted Tomcat this is expected. (2)failed:Server Connection was aborted
   at BonCodeAJP13.BonCodeAJP13ServerConnection.ConnectionError(String message, String messageType)
   at BonCodeAJP13.BonCodeAJP13ServerConnection.ComunicateWithTomcat()
   at BonCodeAJP13.BonCodeAJP13ServerConnection.HandleConnection()
   at BonCodeAJP13.BonCodeAJP13ServerConnection.p_CreateConnection(BonCodeAJP13PacketCollection packetsToSend)

The one that jumps out to me is the System.OutOfMemoryException one. There is nothing in the Boncode docs to address this exception. The Boncode logs had several errors like this, and then had zero entries for the next 40 minutes until it suddenly started up again with connection number 1

2018-12-06 19:51:22 New Connection 1 of 0 to tomcat: [::1]:8009 ID: 1 [T-6]

The exact time Boncode started up again, these errors were in the WIndows event logs:

This warning in the system logs:

A process serving application pool 'XXXXX' suffered a fatal communication error with the Windows Process Activation Service. The process id was '22004'. The data field contains the error number.

And this error in the Application log:

Application: w3wp.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OutOfMemoryException
   at System.Threading.ThreadPoolWorkQueueThreadLocals..ctor(System.Threading.ThreadPoolWorkQueue)
   at System.Threading.ThreadPoolWorkQueue.EnsureCurrentThreadHasQueue()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

There seems to have been a major issue with the application pool and running out of memory but what can I do to troubleshoot this?

[Bilal-S]

Brad,

What is the hardware spec on this machine? What OS are we running exactly? What kind of traffic is this client sending through IIS? Is traffic many upload/download of files?

It looks like that you are running out of memory and have to wait until the App pool memory is recycled before the system becomes responsive again.

Normally, the IIS process buffers traffic in memory. Files, when send without buffers (flushes), will fill up app pool memory at times. Even when IIS releases memory, it takes a while for GC to age out the memory and make it available. This process can be fine tuned based on workloads on the machine. In general, memory issues require a reassessment of how memory and GC is managed. Which boils down to two different strategies: a) faster collection and release of discarded memory b) more memory added to machine or process

If you have enough physical memory you can try to adjust the memory sizes allocated to the app pool. Advanced Settings -> Recycling adjust Private Memory Limit to a higher level

You can tweak GC strategy, I cannot make any direct recommendations without knowledge. Here is a blog on GC algorithms: http://dave-black.blogspot.com/2012/04/how-to-determine-which-garbage.html

Also, you can turn the connector into aggressively releasing memory making explicit GC calls rather than have Windows/IIS manage it using this setting:. <EnableAggressiveGC>

Best, Bilal

[RickGroenewegen]

Hi Bilal,

Brad's client here :) Sorry it took so long to reply, but here is the info you requested:

• Windows Server 2016 Datacenter
• 64-bit OS, x64-based processor
• Intel Xeon CPU E5-2670 v2 @ 2.50Ghz
• Installed memory: 16GB

About the traffic: Only page requests (About 5-10 pages / sec), no big uploads / downloads.

What would you recommend for physical memory sizes?

[Bilal-S]

Thanks Rick. Without much load most of my assumptions become invalid and another element may be involved.

However, have you tried to make the adjustments that I am outlining above, e.g.

• change Privat Memory Limit to higher level
• check what GC strategy you are using and make adjustments, normally workstation strategy is more agressive:

  Adjust GC collection levels to “Workstation GC” model in the .net configuration file. 
  For 64-bit .net 4 this file is located typically in c:\Windows\Microsoft.NET\Framework64\v4.0.30319\Aspnet.config add the line 
  <gcServer enabled="false"/> under <runtime> node

  or use in BonCodeAJP13.settings file?

[Bilal-S]

closing for now.