Edge Hub memory usage and log sizes

[aoeah]

Hello, We have our first installation of the IoT Edge on a server. After a while, the Microsoft.Azure.Devices.Edge.Hub.Service.dll service is using 30GB of memory. Secondly, we were having a lot of problems with the logs growing extremely quickly and filling up the disk. The container logs are growing at about 1 megabyte every 2 seconds or so, if not a little faster. We have 8 devices and a single module. I have a feeling we are doing something incorrectly but nothing is sticking out to me.

I appreciate any guidance!

Here is an excerpt

{"log":"\u003c6\u003e 2021-11-19 22:18:54.528 +00:00 [INF] - Disposing MessagingServiceClient for device Id STNCPU2 because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.MultipleConnectionsException: Multiple connections detected for device STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.528175649Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.528 +00:00 [INF] - Closing device proxy for device Id STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.528439993Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.528 +00:00 [INF] - Client STNCPU2 connected to edgeHub, processing existing subscriptions.\n","stream":"stdout","time":"2021-11-19T22:18:54.528528924Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.529 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.529283282Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.529 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU8.\n","stream":"stdout","time":"2021-11-19T22:18:54.52969226Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.529 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.532648504Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.529 +00:00 [INF] - Processing pending subscriptions for STNCPU8\n","stream":"stdout","time":"2021-11-19T22:18:54.532664148Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.530 +00:00 [INF] - Set subscriptions from session state for STNCPU8\n","stream":"stdout","time":"2021-11-19T22:18:54.532671081Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.530 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU8.\n","stream":"stdout","time":"2021-11-19T22:18:54.532675474Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.530 +00:00 [INF] - Processing pending subscriptions for STNCPU8\n","stream":"stdout","time":"2021-11-19T22:18:54.532692216Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.530 +00:00 [INF] - Set subscriptions from session state for STNCPU8\n","stream":"stdout","time":"2021-11-19T22:18:54.532700402Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.53270472Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Bind device proxy for device STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.532708735Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Binding message channel for device Id STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.532712421Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU2.\n","stream":"stdout","time":"2021-11-19T22:18:54.532724949Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.532728778Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Set subscriptions from session state for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.532732797Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU2.\n","stream":"stdout","time":"2021-11-19T22:18:54.532736758Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.531 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.532740869Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.532 +00:00 [INF] - Set subscriptions from session state for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.532744803Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.532 +00:00 [INF] - Client STNCPU2 in device scope authenticated locally.\n","stream":"stdout","time":"2021-11-19T22:18:54.532748692Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.532 +00:00 [INF] - Successfully generated identity for clientId STNCPU2 and username STNCPUiot01/STNCPU2/?api-version=2020-09-30\u0026DeviceClientType=.NET%2F1.37.2%20%28.NET%20Framework%204.8.4420.0%3B%20Microsoft%20Windows%2010.0.19044%20WindowsProduct%3A0x00000004%3B%20X86%29\n","stream":"stdout","time":"2021-11-19T22:18:54.532850895Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.532 +00:00 [INF] - ClientAuthenticated, STNCPU2, 360ef88c\n","stream":"stdout","time":"2021-11-19T22:18:54.532860743Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.533 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU8.\n","stream":"stdout","time":"2021-11-19T22:18:54.533423355Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.533 +00:00 [INF] - Processing pending subscriptions for STNCPU8\n","stream":"stdout","time":"2021-11-19T22:18:54.533579329Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.534 +00:00 [INF] - Set subscriptions from session state for STNCPU8\n","stream":"stdout","time":"2021-11-19T22:18:54.534104207Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.534 +00:00 [INF] - New device connection for device STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.534349558Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.534 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU2.\n","stream":"stdout","time":"2021-11-19T22:18:54.53482884Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.534 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.535031537Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.535 +00:00 [INF] - Set subscriptions from session state for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.535364162Z"}
{"log":"\u003c4\u003e 2021-11-19 22:18:54.535 +00:00 [WRN] - Closing connection for device: STNCPU2, Microsoft.Azure.Devices.Edge.Hub.Core.MultipleConnectionsException: Multiple connections detected for device STNCPU2, \n","stream":"stdout","time":"2021-11-19T22:18:54.535563376Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.535 +00:00 [INF] - Disposing MessagingServiceClient for device Id STNCPU2 because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.MultipleConnectionsException: Multiple connections detected for device STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.535662522Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.535 +00:00 [INF] - Closing device proxy for device Id STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.536057173Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.536 +00:00 [INF] - Client STNCPU2 connected to edgeHub, processing existing subscriptions.\n","stream":"stdout","time":"2021-11-19T22:18:54.53616768Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.536 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.536533587Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.536 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.536792558Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.537 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.537070512Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.537 +00:00 [INF] - Bind device proxy for device STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.537215211Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.537 +00:00 [INF] - Binding message channel for device Id STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.537329183Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.537 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU2.\n","stream":"stdout","time":"2021-11-19T22:18:54.537484762Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.537 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.537670432Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.537 +00:00 [INF] - Set subscriptions from session state for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.537990803Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.538 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU2.\n","stream":"stdout","time":"2021-11-19T22:18:54.538185017Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.538 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.538357138Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.538 +00:00 [INF] - Set subscriptions from session state for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.538670951Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.541 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU2.\n","stream":"stdout","time":"2021-11-19T22:18:54.544517735Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.541 +00:00 [INF] - Processing pending subscriptions for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.5446072Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.542 +00:00 [INF] - Set subscriptions from session state for STNCPU2\n","stream":"stdout","time":"2021-11-19T22:18:54.544655272Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.545 +00:00 [INF] - Client STNCPU1 in device scope authenticated locally.\n","stream":"stdout","time":"2021-11-19T22:18:54.545913798Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.545 +00:00 [INF] - Successfully generated identity for clientId STNCPU1 and username STNCPUiot01/STNCPU1/?api-version=2020-09-30\u0026DeviceClientType=.NET%2F1.37.2%20%28.NET%20Framework%204.8.4420.0%3B%20Microsoft%20Windows%2010.0.19044%20WindowsProduct%3A0x00000004%3B%20X86%29\n","stream":"stdout","time":"2021-11-19T22:18:54.545928869Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.545 +00:00 [INF] - ClientAuthenticated, STNCPU1, 6004458c\n","stream":"stdout","time":"2021-11-19T22:18:54.54596461Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - New device connection for device STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546214937Z"}
{"log":"\u003c4\u003e 2021-11-19 22:18:54.546 +00:00 [WRN] - Closing connection for device: STNCPU1, Microsoft.Azure.Devices.Edge.Hub.Core.MultipleConnectionsException: Multiple connections detected for device STNCPU1, \n","stream":"stdout","time":"2021-11-19T22:18:54.546246528Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Disposing MessagingServiceClient for device Id STNCPU1 because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.MultipleConnectionsException: Multiple connections detected for device STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546251635Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Closing device proxy for device Id STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546281048Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Client STNCPU1 connected to edgeHub, processing existing subscriptions.\n","stream":"stdout","time":"2021-11-19T22:18:54.546312492Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Bind device proxy for device STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.54651899Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Binding message channel for device Id STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546527734Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Processing pending subscriptions for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546669995Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Processing pending subscriptions for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546803044Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Processing pending subscriptions for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546849514Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.549 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU1.\n","stream":"stdout","time":"2021-11-19T22:18:54.549965451Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.550 +00:00 [INF] - Processing pending subscriptions for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.550376944Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.550 +00:00 [INF] - Set subscriptions from session state for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.551254757Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.550 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU1.\n","stream":"stdout","time":"2021-11-19T22:18:54.551308489Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.550 +00:00 [INF] - Processing pending subscriptions for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.551336863Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.551 +00:00 [INF] - Set subscriptions from session state for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.551399847Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.553 +00:00 [INF] - Processing subscriptions TwinResponse, Methods, C2D for client STNCPU1.\n","stream":"stdout","time":"2021-11-19T22:18:54.553336752Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.553 +00:00 [INF] - Processing pending subscriptions for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.553351371Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.553 +00:00 [INF] - Set subscriptions from session state for STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.553501941Z"}

[nyanzebra]

For problem 1: That seems disconcerting... Could you provide a support bundle (iotedge support-bundle)? This will help us evaluate what is happening. For problem 2: I don't think there are log limits for docker containers, but with RuntimeLogLevel environment variable the log level can be adjusted to avoid logging that might not be of interest to you.

https://github.com/Azure/iotedge/blob/master/doc/EnvironmentVariables.md

[nyanzebra]

^ actually, the support-bundle might share too much information, could you instead just share the result of iotedge check?

[aoeah]

Hello, thank you for your replies! I have saved the output. There are obviously a few warnings. Do any of these point you to a potential cause? I see a few that my operations team should address and I will make sure they do so. Thank you for any help. If I can provide additional details, please let me know.

√ config.yaml is well-formed - OK
√ config.yaml has well-formed connection string - OK
√ container engine is installed and functional - OK
‼ config.yaml has correct hostname - Warning
    config.yaml has hostname XXXXXXX which does not comply with RFC 1035.

    - Hostname must be between 1 and 255 octets inclusive.
    - Each label in the hostname (component separated by ".") must be between 1 and 63 octets inclusive.
    - Each label must start with an ASCII alphabet character (a-z, A-Z), end with an ASCII alphanumeric character (a-z, A-Z, 0-9), and must contain only ASCII alphanumeric characters or hyphens (a-z, A-Z, 0-9, "-").

    Not complying with RFC 1035 may cause errors during the TLS handshake with modules and downstream devices.
√ config.yaml has correct URIs for daemon mgmt endpoint - OK
‼ latest security daemon - Warning
    Installed IoT Edge daemon has version 1.1.4 but 1.1.8 is the latest stable version available.
    Please see https://aka.ms/iotedge-update-runtime for update instructions.
√ host time is close to real time - OK
√ container time is close to host time - OK
‼ DNS server - Warning
    Container engine is not configured with DNS server setting, which may impact connectivity to IoT Hub.
    Please see https://aka.ms/iotedge-prod-checklist-dns for best practices.
    You can ignore this warning if you are setting DNS server per module in the Edge deployment.
√ production readiness: identity certificates expiry - OK
√ production readiness: certificates - OK
√ production readiness: container engine - OK
‼ production readiness: logs policy - Warning
    Container engine is not configured to rotate module logs which may cause it run out of disk space.
    Please see https://aka.ms/iotedge-prod-checklist-logs for best practices.
    You can ignore this warning if you are setting log policy per module in the Edge deployment.
‼ production readiness: Edge Agent's storage directory is persisted on the host filesystem - Warning
    The edgeAgent module is not configured to persist its /tmp/edgeAgent directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.
‼ production readiness: Edge Hub's storage directory is persisted on the host filesystem - Warning
    The edgeHub module is not configured to persist its /tmp/edgeHub directory on the host filesystem.
    Data might be lost if the module is deleted or updated.
    Please see https://aka.ms/iotedge-storage-host for best practices.

Connectivity checks
-------------------
√ host can connect to and perform TLS handshake with IoT Hub AMQP port - OK
√ host can connect to and perform TLS handshake with IoT Hub HTTPS / WebSockets port - OK
√ host can connect to and perform TLS handshake with IoT Hub MQTT port - OK
√ container on the default network can connect to IoT Hub AMQP port - OK
√ container on the default network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the default network can connect to IoT Hub MQTT port - OK
√ container on the IoT Edge module network can connect to IoT Hub AMQP port - OK
√ container on the IoT Edge module network can connect to IoT Hub HTTPS / WebSockets port - OK
√ container on the IoT Edge module network can connect to IoT Hub MQTT port - OK

18 check(s) succeeded.
6 check(s) raised warnings. Re-run with --verbose for more details.

[nyanzebra]

So, not sure if this will solve the problem, but you should be able to follow https://docs.microsoft.com/en-us/azure/iot-edge/production-checklist?view=iotedge-2020-11#place-limits-on-log-size for limiting some of the issue you are seeing (log growth).

Secondly, do you have more than one device using the same identifier? This shouldn't cause the memory to explode, but is still something you might want to check to make sure all your devices are properly sending messages to the broker.

{"log":"\u003c4\u003e 2021-11-19 22:18:54.546 +00:00 [WRN] - Closing connection for device: STNCPU1, Microsoft.Azure.Devices.Edge.Hub.Core.MultipleConnectionsException: Multiple connections detected for device STNCPU1, \n","stream":"stdout","time":"2021-11-19T22:18:54.546246528Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Disposing MessagingServiceClient for device Id STNCPU1 because of exception - Microsoft.Azure.Devices.Edge.Hub.Core.MultipleConnectionsException: Multiple connections detected for device STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546251635Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Closing device proxy for device Id STNCPU1\n","stream":"stdout","time":"2021-11-19T22:18:54.546281048Z"}
{"log":"\u003c6\u003e 2021-11-19 22:18:54.546 +00:00 [INF] - Client STNCPU1 connected to edgeHub, processing existing subscriptions.\n","stream":"stdout","time":"2021-11-19T22:18:54.546312492Z"}

[nyanzebra]

Also, on the topic of memory usage, about how much traffic are you sending through the broker per second? I would like to replicate your situation as closely as possible and see if there is a memory leak. I am also assuming the flow of messages is something like this:

8 devices registered to send upstream through a module to EH?
1 ---\
2 ---\
          Module-->EdgeHub--> upstream to cloud
3 ---/
4 ---/
...

[aoeah]

Thank you again for all of your help @nyanzebra! I do have an idea in regards to why there are some duplicates.

Our use-case is a little bit different. We do have some messages flowing upstream, however, most messages are sourced from some sensors we have and routed to a module. The module processes and sends a result to some other devices. We weren't able to use the built-in MQTT because it wasn't GA when initially developing the solution so we leveraged direct messages for best-attempt delivery.

In terms of traffic per second, it isn't substantial. We have bursts that could be ~10 in a second, however, typically we are looking at about 1 per second.

Here is a rough diagram that illustrates it

[aoeah]

@nyanzebra Based on your observations with the log issue, I was able to reproduce the issue. Both the logging and the memory are related. Your observation that we had two devices with the same name was correct.

For your own knowledge. The way that our system works, on the devices indicated as Device 1, 2 and 3, there are actually two applications running. Each of those applications are identifying itself with the same device id. As soon as both of those applications were running, it was actually causing a battle of connections. It seems that the server sees a duplicate connection and then drops one of them. The client application then seems to identify it was disconnected and reconnects which, in turn, seems to cause the server to drop one of the connections again.

We were able to test in an environment without any activity. As soon as we started up both applications we started to see the logs going frantically. We also saw that the CPU usage increased quite a bit as well as an incrementing memory footprint. Note that we were not sending any messages or anything. The increasing memory footprint was only just having the applications connecting over and over and over again. This seems like it could be a possible problem in the code base, however, people shouldn't be doing this anyway =)

Thank you again for your help!