Closed ccruden-aspire closed 9 months ago
On a hunch, I tried replacing the DotNetty 0.7.0 assemblies included with MADC 1.40 with DotNetty 0.7.5 assemblies, and that appears to have resolved the main problem in this issue - the giant memory consumption. The issue where the service stops and starts sending regularly is still present, but that's probably IoT hub related...
I'm encountering a similar issue when trying to connect a device using OpenAsync() that is disabled in the IoT Hub. The process then allocates all available memory in almost no time.
I'm on
Updating to DotNetty 0.7.5 seems to have fixed that issue, thanks @ccruden-aspire!
We've upgraded the dotnetty dependency to the latest in #3401, so I'll mark this as fix checked in
This issue has been fixed as of this release, so I'll close this thread
Context
Description of the issue
We have a service whose only job is to read messages from files in a directory and write the contents of those files to IoT hub messages. It has been in service at dozens of sites for several years transferring gigabytes a day without seeing this problem. However, starting a day ago, it seems two IoT hubs have entered a state which seems to cause a message sent with SendEventAsync to block / loop, and consume all available memory. In seconds the process goes from using 70M of memory to 25G and either has to be killed off or the machine rebooted. This only seems to occur under particular circumstances:
Code sample exhibiting the issue
It's impossible to isolate this to code since the issue is only happening at a couple of production sites and we don't have access to experiment with those IoT hubs. Relevant code included below. The service involves pulling from a BlockingCollection of IQueueMessages and awaiting a call to SendToHub.
Console log of the issue
The service's log at Debug level ends on a line with "Sent # byte message to IoTHub" several seconds before the process had to be killed off. Before that, that message was occurring multiple times a second. adc_000001.zip