search

Azure / iotedge

The IoT Edge OSS project
MIT License
1.45k stars 458 forks source link

Memory Leak in edgeHub module #821

Closed BewaAutomatisierung-RD closed 4 years ago

BewaAutomatisierung-RD commented 5 years ago

Expected Behavior

The edgeHub module should be able to run for a long time and receive tons of messages without increasing memory consumption.

Current Behavior

Memory consumption of the edgeHub module increases constantly if messages are handled. Before Handling messages the memory consumption of edgeHub is about 115 MB. After 10.000 messages generated by tempSensorModule and passed through a SampleModule just piping the messages to IotHub, the memory consumption of edgeHub is about 150 MB. It does not decrease again after stopping sending messages but stays the same. Memory consumption of tempSensor and SampleModule increase just a little bit temporarily but will always decrease again.

The increasing Memory consumption occurs on both a Raspberry Pi and an ARM64 Virtual Machine on Azure. On the Raspberry Pi the edgeHub crashes at some point with std::badAlloc when the address space is exceeded.

Steps to Reproduce

  1. Create new IotEdge Solution in Visual Studio Code using the C# Module template
  2. Set MessageCount of tempSensor to 10000 and MessageDelay to "00:00:00.010"
  3. Deploy to Device

Context (Environment)

Device (Host) Operating System

Architecture

Container Operating System

Runtime Versions

iotedged

Edge Agent

Edge Hub

Docker

Logs

Additional Information

darobs commented 5 years ago

Hello @BewaControl-ReneDivossen

Some of that memory is probably pooled buffer allocation. The EdgeHub will allocate larger and larger pools of buffer memory as queues fill. I am not certain when or if those memory pools will be garbage collected, because it may continue to use these larger pools for quite a while.

One thing you definitely do not want to do is let EdgeHub allocate very large buffer pools on the Raspberry Pi. If you are not setting "OptimizeForPerformance" to false on Pi deployments, you may see EdgeHub try to allocate large chunks of memory the system cannot accommodate. Please double-check that.

So, I was talking with my teammates, and we have some ideas which could alleviate memory pressure. We're going to follow up with the dotnet team to see if there are memory management options we can set, especially for constrained systems like the Pi.

However, it's also possible that we have a memory leak - so I'm going to put a task to check for memory leaks in our backlog.

BewaAutomatisierung-RD commented 5 years ago

Hello @darobs

The "OptimizeForPerformance" setting is set to false (on both the Raspberry and the VirtualMachine). We also use a custom storage path for messages. Here is the deployment setting for the sample application for reproduction of the issue:

{
  "modulesContent": {
    "$edgeAgent": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "runtime": {
          "type": "docker",
          "settings": {
            "minDockerVersion": "v1.25",
            "loggingOptions": "",
            "registryCredentials": {
              // removed from posting
              }
            }
          }
        },
        "systemModules": {
          "edgeAgent": {
            "type": "docker",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-agent:1.0",
              "createOptions": "{}"
            }
          },
          "edgeHub": {
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-hub:1.0",
              "createOptions": "{\"HostConfig\":{\"Binds\":[\"/etc/iotedge/storage/:/iotedge/storage/\"],\"PortBindings\":{\"5671/tcp\":[{\"HostPort\":\"5671\"}],\"8883/tcp\":[{\"HostPort\":\"8883\"}],\"443/tcp\":[{\"HostPort\":\"443\"}]}}}"
            },
            "env": {
              "OptimizeForPerformance": {
                "value": "false"
              },
              "storageFolder": {
                "value": "/iotedge/storage/"
              }
            }
          }
        },
        "modules": {
          "tempSensor": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "mcr.microsoft.com/azureiotedge-simulated-temperature-sensor:1.0",
              "createOptions": "{}"
            },
            "env": {
              "MessageCount": {
                "value": 10000
              },
              "MessageDelay": {
                "value": "00:00:00.010"
              }
            }
          },
          "SampleModule": {
            "version": "1.0",
            "type": "docker",
            "status": "running",
            "restartPolicy": "always",
            "settings": {
              "image": "bewaconnect.azurecr.io/samplemodule:0.0.1-amd64",
              "createOptions": "{}"
            }
          }
        }
      }
    },
    "$edgeHub": {
      "properties.desired": {
        "schemaVersion": "1.0",
        "routes": {
          "SampleModuleToIoTHub": "FROM /messages/modules/SampleModule/outputs/* INTO $upstream",
          "sensorToSampleModule": "FROM /messages/modules/tempSensor/outputs/temperatureOutput INTO BrokeredEndpoint(\"/modules/SampleModule/inputs/input1\")"
        },
        "storeAndForwardConfiguration": {
          "timeToLiveSecs": 7200
        }
      }
    }
  }
}

We are reaching a point where we have to question whether IotEdge can really be used in real life industrial use-cases. Our machines will be sending messages every 2 seconds and they have to be able to run 24/7 for long periods of time without worrying about loosing data. I'm under time pressure to get the messages to IotHub reliably so it would be great to get an estimation soon, if there is a way to use iotedge in real-world industry scenarios and if yes, which hardware and configuration has to be used to make it work.

I just found a report from March 18 stating there probably is a memory leak: https://github.com/Azure/iot-edge-v1/issues/537

Please let me know as soon as possible if there really is an issue and if it can be fixed soon. If it can't I'll have to look for an alternative. I'm still hoping that there is a configuration that makes it all work!

Thanks, René

BewaAutomatisierung-RD commented 5 years ago

According to a recommendation by @varunpuranik in this thread https://github.com/Azure/iotedge/issues/119 i turned off AMQP. The EdgeHub then starts with about 57 MB of memory consumption and after 10000 messages consumes about 97 MB. This means about 4KB per Message are allocated and never released. This is about the same as with AMQP and MQTT both turned on (increase from about 115 MB to 150 MB).
I also did another test with MQTT turned off and only using AMQP. The memory consumption increased as well. It just started higher (at about 106 MB).

varunpuranik commented 5 years ago

@BewaControl-ReneDivossen - We have long haul tests that run fine with workloads similar to what you have for several days (we run each test for 7 days). We have also done testing where we have been able to run IoT Edge for weeks without any issues under similar loads. As for the memory consumption - the memory usage of EdgeHub when both protocols are enabled typically puts a lot of pressure on a device like Raspberry PI. But with one protocol head turned off, the memory pressure goes down as you see. The increase in memory usage is not necessarily a leak, but just the way the Dotnet GC behaves, allowing the app to consume more memory when available. But the expectation is that this should stabilize at some point, after which the EdgeHub should be able to run steadily for long periods of time. It would be great if you can test this. Meanwhile as @darobs suggested, we will once again check EdgeHub for any memory leaks. We are also looking at tweaking certain GC settings to see if we can get an even better and more reliable performance from EdgeHub.

myagley commented 5 years ago

Are you actually seeing issues with message loss, errors, etc? Or, is this just a comment on the rising memory use? If there are issues with message loss or other errors, is it possible to get more detail?

As @darobs mentions above, the Edge Hub is implemented with .NET, which is a managed runtime with a garbage collector. Its memory use will not look like a native process with manual memory mangement. Rising memory in the Edge Hub does not mean a memory leak. As you mention, there was a true memory leak in the Edge Hub at the beginning of 2018, but this has been fixed.

There is an issue on constrained devices with 32 bit address space with a bad_alloc and we are looking at ways to tune the dotnet garbage collection to optimize for memory footprint instead of collection performance on these platforms. This means collecting more often which has negative performance impact.

If there are real concerns about the Edge Hub using all of the memory on the device, you can constrain its memory use in the createOptions when deploying. This configuration can be provided if you are interested. With your message rate, you should be prepared to give the Edge Hub at least 1Gb of memory.

BewaAutomatisierung-RD commented 5 years ago

With turning off protocol heads as described by @varunpuranik in this thread it seems to work alright. Memory is rising for a while but then seems to stabilize at some point. Short term tests with only MQTT enabled (3 days) and only AMQP enabled (1 day) produced no more bad_alloc crashes. If I see any further problems when doing long term tests, I'll get back to you. I'll also keep this thread subscribed to see if you find any more memory issues and/or if you find a way to get rid of the bad_alloc errors even with all protocol heads enabled.

I also mentioned this in the other thread but I'll copy this here: To prevent other developers from becoming frustrated you should comment on the necessity/recommendation to turn off Protocol heads on arm32 devices in your Installation Guide on your Troubleshooting Page. So far Turning off Protocol Heads seems to be an undocumented feature.

veyalla commented 5 years ago

Thank you for the feedback. The turning off of protocol heads is documented in the production guide: https://docs.microsoft.com/en-us/azure/iot-edge/production-checklist#disable-unused-protocols

If you're concerned about resource usage, it is also possible to set memory limits on the container via module create options.

I agree that this would be good info to have in the troubleshooting section for resource constrained devices. I'll update the page.

levi106 commented 5 years ago

I got a similar problem when I disabled outbound network to IoT Hub. If my understanding is correct, Visual Studio shows only live objects, so these objects will not be collected by GC.

appinsights dump

It seems that one of the causes of the memory leak is that the event handler ConnectivityAwareClient.HandleDeviceConnectedEvent will not be removed from DeviceConnectivityManager.DeviceConnected.

0:000> !gcroot  1294d650b28
Thread 22b4:
    00000010b3d7dcf0 00007ff9deb504b7 System.Threading.Tasks.Task.SpinThenBlockingWait(Int32, System.Threading.CancellationToken) [E:\A\_work\62\s\src\mscorlib\src\System\Threading\Tasks\Task.cs @ 2978]
        rbp+10: 00000010b3d7dd60
            ->  000001294d0bba08 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[System.Int32, System.Private.CoreLib],[Microsoft.Azure.Devices.Edge.Hub.Service.Program+<MainAsync>d__2, Microsoft.Azure.Devices.Edge.Hub.Service]]
            ->  000001294d243c98 Microsoft.Azure.Devices.Edge.Hub.Service.Hosting
            ->  000001294d172298 Autofac.Core.Container
            ->  000001294d172920 Autofac.Core.Lifetime.LifetimeScope
            ->  000001294d1729a0 System.Collections.Generic.Dictionary`2[[System.Guid, System.Private.CoreLib],[System.Object, System.Private.CoreLib]]
            ->  000001294d4ef088 System.Collections.Generic.Dictionary`2+Entry[[System.Guid, System.Private.CoreLib],[System.Object, System.Private.CoreLib]][]
            ->  000001294d23b6e0 Microsoft.Azure.Devices.Edge.Storage.RocksDb.DbStoreProvider
            ->  000001294d23b9c8 System.Threading.Timer
            ->  000001294d23ba58 System.Threading.TimerHolder
            ->  000001294d23b9f8 System.Threading.TimerQueueTimer
            ->  000001294d0b5628 System.Threading.TimerQueue
            ->  000001294d0e5c68 System.Threading.TimerQueueTimer
            ->  000001295175aad0 System.Threading.TimerQueueTimer
            ->  00000129516812a0 System.Threading.TimerQueueTimer
            ->  0000012951680260 System.Threading.TimerQueueTimer
            ->  000001295167df68 System.Threading.TimerQueueTimer
            ->  0000012951679f00 System.Threading.TimerQueueTimer
            ->  00000129512a4ab0 System.Threading.TimerQueueTimer
            ->  00000129512a4200 System.Threading.TimerQueueTimer
            ->  00000129512a3950 System.Threading.TimerQueueTimer
            ->  0000012951212aa0 System.Threading.TimerQueueTimer
            ->  000001295120cfa8 System.Threading.TimerQueueTimer
            ->  000001295120b9c8 System.Threading.TimerQueueTimer
            ->  00000129511758c8 System.Threading.TimerQueueTimer
            ->  0000012951174ff8 System.Threading.TimerQueueTimer
            ->  0000012951174728 System.Threading.TimerQueueTimer
            ->  000001294de5a788 System.Threading.TimerQueueTimer
            ->  000001294de54c78 System.Threading.TimerQueueTimer
            ->  000001294de543c8 System.Threading.TimerQueueTimer
            ->  000001294ddc17c8 System.Threading.TimerQueueTimer
            ->  000001294ddc0ef8 System.Threading.TimerQueueTimer
            ->  000001294ddbe048 System.Threading.TimerQueueTimer
            ->  000001295162c968 System.Threading.TimerQueueTimer
            ->  000001295162bdb0 System.Threading.TimerQueueTimer
            ->  000001295162a7d8 System.Threading.TimerQueueTimer
            ->  000001294dd4cf20 System.Threading.TimerQueueTimer
            ->  000001294dd4c650 System.Threading.TimerQueueTimer
            ->  000001294dd48630 System.Threading.TimerQueueTimer
            ->  000001294dcb4248 System.Threading.TimerQueueTimer
            ->  000001294dcb3998 System.Threading.TimerQueueTimer
            ->  000001294dcb23b8 System.Threading.TimerQueueTimer
            ->  000001294dc1e968 System.Threading.TimerQueueTimer
            ->  000001294dc1af48 System.Threading.TimerQueueTimer
            ->  000001294dc187b8 System.Threading.TimerQueueTimer
            ->  000001294db87328 System.Threading.TimerQueueTimer
            ->  000001294db86a78 System.Threading.TimerQueueTimer
            ->  000001294db86180 System.Threading.TimerQueueTimer
            ->  000001294daf2110 System.Threading.TimerQueueTimer
            ->  000001294daee0f0 System.Threading.TimerQueueTimer
            ->  000001294daecb18 System.Threading.TimerQueueTimer
            ->  000001294da4fc98 System.Threading.TimerQueueTimer
            ->  000001294da4ef58 System.Threading.TimerQueueTimer
            ->  000001294da4d960 System.Threading.TimerQueueTimer
            ->  0000012951596a08 System.Threading.TimerQueueTimer
            ->  0000012951596158 System.Threading.TimerQueueTimer
            ->  00000129515958a8 System.Threading.TimerQueueTimer
            ->  00000129515061e8 System.Threading.TimerQueueTimer
            ->  0000012951505918 System.Threading.TimerQueueTimer
            ->  00000129515011d8 System.Threading.TimerQueueTimer
            ->  000001294d9aac60 System.Threading.TimerQueueTimer
            ->  000001294d9aa3b0 System.Threading.TimerQueueTimer
            ->  000001294d9a9b00 System.Threading.TimerQueueTimer
            ->  000001294d918a08 System.Threading.TimerQueueTimer
            ->  000001294d916878 System.Threading.TimerQueueTimer
            ->  000001294d9122c0 System.Threading.TimerQueueTimer
            ->  000001294d8799c0 System.Threading.TimerQueueTimer
            ->  000001294d879110 System.Threading.TimerQueueTimer
            ->  000001294d877b30 System.Threading.TimerQueueTimer
            ->  000001294d7e21b0 System.Threading.TimerQueueTimer
            ->  000001294d7de188 System.Threading.TimerQueueTimer
            ->  000001294d7dd8d8 System.Threading.TimerQueueTimer
            ->  000001294d74b6d8 System.Threading.TimerQueueTimer
            ->  000001294d74ae28 System.Threading.TimerQueueTimer
            ->  000001294d74a578 System.Threading.TimerQueueTimer
            ->  000001294d6b1c60 System.Threading.TimerQueueTimer
            ->  000001294d6af2b8 System.Threading.TimerQueueTimer
            ->  000001294d6aea08 System.Threading.TimerQueueTimer
            ->  000001294d61df20 System.Threading.TimerQueueTimer
            ->  000001294d61d670 System.Threading.TimerQueueTimer
            ->  000001294d619c38 System.Threading.TimerQueueTimer
            ->  0000012951474f88 System.Threading.TimerQueueTimer
            ->  00000129514746d8 System.Threading.TimerQueueTimer
            ->  0000012951473e28 System.Threading.TimerQueueTimer
            ->  00000129510eae70 System.Threading.TimerQueueTimer
            ->  0000012951434438 System.Threading.TimerQueueTimer
            ->  0000012951433b88 System.Threading.TimerQueueTimer
            ->  00000129514332d8 System.Threading.TimerQueueTimer
            ->  0000012951431c10 System.Threading.TimerQueueTimer
            ->  0000012951431360 System.Threading.TimerQueueTimer
            ->  0000012951430ab0 System.Threading.TimerQueueTimer
            ->  0000012951430048 System.Threading.TimerQueueTimer
            ->  000001295142f798 System.Threading.TimerQueueTimer
            ->  000001295142eee8 System.Threading.TimerQueueTimer
            ->  000001295142e518 System.Threading.TimerQueueTimer
            ->  000001295142dc68 System.Threading.TimerQueueTimer
            ->  000001295142d3b8 System.Threading.TimerQueueTimer
            ->  000001295142c950 System.Threading.TimerQueueTimer
            ->  000001295142c0a0 System.Threading.TimerQueueTimer
            ->  000001295142b7f0 System.Threading.TimerQueueTimer
            ->  000001295142aeb8 System.Threading.TimerQueueTimer
            ->  000001295142a608 System.Threading.TimerQueueTimer
            ->  000001294d58b660 System.Threading.TimerQueueTimer
            ->  000001294d5888e0 System.Threading.TimerQueueTimer
            ->  000001294d588030 System.Threading.TimerQueueTimer
            ->  000001294d587780 System.Threading.TimerQueueTimer
            ->  000001294d586db0 System.Threading.TimerQueueTimer
            ->  0000012951425e30 System.Threading.TimerQueueTimer
            ->  000001294d584af8 System.Threading.TimerQueueTimer
            ->  0000012951428048 System.Threading.TimerQueueTimer
            ->  0000012951427778 System.Threading.TimerQueueTimer
            ->  0000012951426e68 System.Threading.TimerQueueTimer
            ->  00000129514241e0 System.Threading.TimerQueueTimer
            ->  0000012951423930 System.Threading.TimerQueueTimer
            ->  0000012951423080 System.Threading.TimerQueueTimer
            ->  0000012951422220 System.Threading.TimerQueueTimer
            ->  0000012951421970 System.Threading.TimerQueueTimer
            ->  00000129514210c0 System.Threading.TimerQueueTimer
            ->  0000012951420308 System.Threading.TimerQueueTimer
            ->  000001295141fa58 System.Threading.TimerQueueTimer
            ->  000001295141f1a8 System.Threading.TimerQueueTimer
            ->  000001295141e2f0 System.Threading.TimerQueueTimer
            ->  000001295141da40 System.Threading.TimerQueueTimer
            ->  000001295141d190 System.Threading.TimerQueueTimer
            ->  000001295141bda0 System.Threading.TimerQueueTimer
            ->  000001295141b4f0 System.Threading.TimerQueueTimer
            ->  0000012951419e78 System.Threading.TimerQueueTimer
            ->  000001294d58ad88 System.Threading.TimerQueueTimer
            ->  000001294d58a410 System.Threading.TimerQueueTimer
            ->  000001294d589a80 System.Threading.TimerQueueTimer
            ->  000001294d584110 System.Threading.TimerQueueTimer
            ->  000001294d583860 System.Threading.TimerQueueTimer
            ->  000001294d582fb0 System.Threading.TimerQueueTimer
            ->  000001294d581d18 System.Threading.TimerQueueTimer
            ->  000001294d581468 System.Threading.TimerQueueTimer
            ->  000001294d580bb8 System.Threading.TimerQueueTimer
            ->  000001294d580040 System.Threading.TimerQueueTimer
            ->  000001294d57f790 System.Threading.TimerQueueTimer
            ->  000001294d57eee0 System.Threading.TimerQueueTimer
            ->  000001294d57e2f8 System.Threading.TimerQueueTimer
            ->  000001294d57da48 System.Threading.TimerQueueTimer
            ->  000001294d57d198 System.Threading.TimerQueueTimer
            ->  000001294d57c600 System.Threading.TimerQueueTimer
            ->  000001294d57bd50 System.Threading.TimerQueueTimer
            ->  000001294d57b4a0 System.Threading.TimerQueueTimer
            ->  000001294d57a4b8 System.Threading.TimerQueueTimer
            ->  000001294d579c08 System.Threading.TimerQueueTimer
            ->  000001294d579358 System.Threading.TimerQueueTimer
            ->  000001294d578818 System.Threading.TimerQueueTimer
            ->  000001294d577f68 System.Threading.TimerQueueTimer
            ->  000001294d5776b8 System.Threading.TimerQueueTimer
            ->  000001294d576668 System.Threading.TimerQueueTimer
            ->  000001294d575db8 System.Threading.TimerQueueTimer
            ->  000001294d575508 System.Threading.TimerQueueTimer
            ->  000001294d574678 System.Threading.TimerQueueTimer
            ->  000001294d573dc8 System.Threading.TimerQueueTimer
            ->  000001294d573518 System.Threading.TimerQueueTimer
            ->  000001294d572660 System.Threading.TimerQueueTimer
            ->  000001294d571db0 System.Threading.TimerQueueTimer
            ->  000001294d571500 System.Threading.TimerQueueTimer
            ->  000001294d570bc8 System.Threading.TimerQueueTimer
            ->  000001294d570318 System.Threading.TimerQueueTimer
            ->  000001294d56fa68 System.Threading.TimerQueueTimer
            ->  000001294d56f130 System.Threading.TimerQueueTimer
            ->  000001294d56e880 System.Threading.TimerQueueTimer
            ->  000001294d56dfd0 System.Threading.TimerQueueTimer
            ->  000001294d56d698 System.Threading.TimerQueueTimer
            ->  000001294d56cde8 System.Threading.TimerQueueTimer
            ->  000001294d56c538 System.Threading.TimerQueueTimer
            ->  000001294d56bc00 System.Threading.TimerQueueTimer
            ->  000001294d56b350 System.Threading.TimerQueueTimer
            ->  000001294d56aaa0 System.Threading.TimerQueueTimer
            ->  000001294d56a168 System.Threading.TimerQueueTimer
            ->  000001294d5698b8 System.Threading.TimerQueueTimer
            ->  000001294d569008 System.Threading.TimerQueueTimer
            ->  000001294d5686d0 System.Threading.TimerQueueTimer
            ->  000001294d567e20 System.Threading.TimerQueueTimer
            ->  000001294d567570 System.Threading.TimerQueueTimer
            ->  000001294d566ae8 System.Threading.TimerQueueTimer
            ->  000001294d566238 System.Threading.TimerQueueTimer
            ->  000001294d565988 System.Threading.TimerQueueTimer
            ->  000001294d5651d0 System.Threading.TimerQueueTimer
            ->  000001294d564920 System.Threading.TimerQueueTimer
            ->  000001294d564070 System.Threading.TimerQueueTimer
            ->  000001294d563738 System.Threading.TimerQueueTimer
            ->  000001294d562e88 System.Threading.TimerQueueTimer
            ->  000001294d5625d8 System.Threading.TimerQueueTimer
            ->  000001294d561ca0 System.Threading.TimerQueueTimer
            ->  000001294d5613f0 System.Threading.TimerQueueTimer
            ->  000001294d560b40 System.Threading.TimerQueueTimer
            ->  000001294d560208 System.Threading.TimerQueueTimer
            ->  000001294d55f958 System.Threading.TimerQueueTimer
            ->  000001294d55f0a8 System.Threading.TimerQueueTimer
            ->  000001294d55e770 System.Threading.TimerQueueTimer
            ->  000001294d55dec0 System.Threading.TimerQueueTimer
            ->  000001294d55d610 System.Threading.TimerQueueTimer
            ->  000001294d55ccd8 System.Threading.TimerQueueTimer
            ->  000001294d55c428 System.Threading.TimerQueueTimer
            ->  000001294d55bb78 System.Threading.TimerQueueTimer
            ->  000001294d55b240 System.Threading.TimerQueueTimer
            ->  000001294d55a990 System.Threading.TimerQueueTimer
            ->  000001294d55a0e0 System.Threading.TimerQueueTimer
            ->  000001294d5597a8 System.Threading.TimerQueueTimer
            ->  000001294d558ef8 System.Threading.TimerQueueTimer
            ->  000001294d558648 System.Threading.TimerQueueTimer
            ->  000001294d557d10 System.Threading.TimerQueueTimer
            ->  000001294d557460 System.Threading.TimerQueueTimer
            ->  000001294d5569f0 System.Threading.TimerQueueTimer
            ->  000001294d5560b8 System.Threading.TimerQueueTimer
            ->  000001294d555808 System.Threading.TimerQueueTimer
            ->  000001294d554f58 System.Threading.TimerQueueTimer
            ->  000001294d554620 System.Threading.TimerQueueTimer
            ->  000001294d553d70 System.Threading.TimerQueueTimer
            ->  000001294d5534c0 System.Threading.TimerQueueTimer
            ->  000001294d552b88 System.Threading.TimerQueueTimer
            ->  000001294d5522d8 System.Threading.TimerQueueTimer
            ->  000001294d551a28 System.Threading.TimerQueueTimer
            ->  000001294d5510b8 System.Threading.TimerQueueTimer
            ->  000001294d550808 System.Threading.TimerQueueTimer
            ->  000001294d54ff58 System.Threading.TimerQueueTimer
            ->  000001294d54f620 System.Threading.TimerQueueTimer
            ->  000001294d54ed70 System.Threading.TimerQueueTimer
            ->  000001294d54e4c0 System.Threading.TimerQueueTimer
            ->  000001294d54db88 System.Threading.TimerQueueTimer
            ->  000001294d54d2d8 System.Threading.TimerQueueTimer
            ->  000001294d54ca28 System.Threading.TimerQueueTimer
            ->  000001294d54c058 System.Threading.TimerQueueTimer
            ->  000001294d54b7a8 System.Threading.TimerQueueTimer
            ->  000001294d54aef8 System.Threading.TimerQueueTimer
            ->  000001294d54a5c0 System.Threading.TimerQueueTimer
            ->  000001294d549d10 System.Threading.TimerQueueTimer
            ->  000001294d549460 System.Threading.TimerQueueTimer
            ->  000001294d5473d0 System.Threading.TimerQueueTimer
            ->  000001294d546b20 System.Threading.TimerQueueTimer
            ->  000001294d546270 System.Threading.TimerQueueTimer
            ->  000001294d545938 System.Threading.TimerQueueTimer
            ->  000001294d545088 System.Threading.TimerQueueTimer
            ->  000001294d544688 System.Threading.TimerQueueTimer
            ->  000001294d5424f8 System.Threading.TimerQueueTimer
            ->  000001294d541c48 System.Threading.TimerQueueTimer
            ->  000001294d53d570 System.Threading.TimerQueueTimer
            ->  000001294d53c060 System.Threading.TimerQueueTimer
            ->  000001294d53b2f8 System.Threading.TimerQueueTimer
            ->  000001294d1bb1c8 System.Threading.TimerQueueTimer
            ->  000001294d406880 System.Threading.TimerQueueTimer
            ->  000001294d3a3248 System.Threading.TimerQueueTimer
            ->  000001294d2971e8 System.Threading.TimerQueueTimer
            ->  000001294d24a6d8 System.Threading.TimerQueueTimer
            ->  000001294d24a680 System.Threading.TimerCallback
            ->  000001294d2493b0 Microsoft.Azure.Devices.Edge.Hub.Core.DeviceScopeIdentitiesCache
            ->  000001294d3a2ef8 System.EventHandler`1[[System.String, System.Private.CoreLib]]
            ->  000001294d3a2cf0 Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionReauthenticator
            ->  000001294d24f718 Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager
            ->  000001294d24f848 System.Collections.Concurrent.ConcurrentDictionary`2[[System.String, System.Private.CoreLib],[Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager+ConnectedDevice, Microsoft.Azure.Devices.Edge.Hub.Core]]
            ->  000001294d24f9d8 System.Collections.Concurrent.ConcurrentDictionary`2+Tables[[System.String, System.Private.CoreLib],[Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager+ConnectedDevice, Microsoft.Azure.Devices.Edge.Hub.Core]]
            ->  000001294d24f8c8 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, System.Private.CoreLib],[Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager+ConnectedDevice, Microsoft.Azure.Devices.Edge.Hub.Core]][]
            ->  000001294d2580c0 System.Collections.Concurrent.ConcurrentDictionary`2+Node[[System.String, System.Private.CoreLib],[Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager+ConnectedDevice, Microsoft.Azure.Devices.Edge.Hub.Core]]
            ->  000001294d257f88 Microsoft.Azure.Devices.Edge.Hub.Core.ConnectionManager+ConnectedDevice
            ->  000001295176de50 System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1+AsyncStateMachineBox`1[[Microsoft.Azure.Devices.Edge.Util.Try`1[[Microsoft.Azure.Devices.Edge.Hub.Core.Cloud.ICloudConnection, Microsoft.Azure.Devices.Edge.Hub.Core]], Microsoft.Azure.Devices.Edge.Util],[Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider+<Connect>d__15, Microsoft.Azure.Devices.Edge.Hub.CloudProxy]]
            ->  000001294d24f3b8 Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnectionProvider
            ->  000001294d249e28 Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClientProvider
            ->  000001294d24a4c0 Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager
            ->  0000012951807dc0 System.EventHandler
            ->  00000129516e2748 System.Object[]
            ->  000001294d650780 System.EventHandler
            ->  000001294d64fb10 Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ConnectivityAwareClient
            ->  000001294d64fbf0 Microsoft.Azure.Devices.Edge.Hub.CloudProxy.ModuleClientWrapper
            ->  000001294d64fc10 Microsoft.Azure.Devices.Client.ModuleClient
            ->  000001294d650158 Microsoft.Azure.Devices.Client.InternalClient
            ->  000001294d6505a8 Microsoft.Azure.Devices.Client.Transport.RetryDelegatingHandler
            ->  000001294d650578 Microsoft.Azure.Devices.Client.Transport.ErrorDelegatingHandler
            ->  000001294d6504e0 Microsoft.Azure.Devices.Client.Transport.ProtocolRoutingDelegatingHandler
            ->  000001294d650b28 Microsoft.Azure.Devices.Client.Transport.Amqp.AmqpTransportHandler

Found 1 unique roots (run '!gcroot -all' to see all roots).
0:000> !do 000001294d24a4c0
Name:        Microsoft.Azure.Devices.Edge.Hub.CloudProxy.DeviceConnectivityManager
MethodTable: 00007ff98a6fcb30
EEClass:     00007ff98a73cef8
Size:        80(0x50) bytes
File:        C:\app\Microsoft.Azure.Devices.Edge.Hub.CloudProxy.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ff98b167e48  400003e        8 ....Hub.CloudProxy]]  0 instance 000001294d24b500 machine
00007ff98a763998  400003f       10  System.Timers.Timer  0 instance 000001294d24b350 connectedTimer
00007ff98a763998  4000040       18  System.Timers.Timer  0 instance 000001294d24b3e8 disconnectedTimer
00007ff98b149208  4000041       20 ...dentity.IIdentity  0 instance 000001294d24a3d8 testClientIdentity
00007ff98a6fc9d8  4000042       40         System.Int32  1 instance                2 state
00007ff98b1f1750  4000043       28 ...nnectivityChecker  0 instance 000001294d256ea0 connectivityChecker
00007ff9dee45f30  4000044       30  System.EventHandler  0 instance 0000012951807dc0 DeviceConnected
00007ff9dee45f30  4000045       38  System.EventHandler  0 instance 0000012951807e40 DeviceDisconnected
0:000> !do 0000012951807dc0
Name:        System.EventHandler
MethodTable: 00007ff9dee45f30
EEClass:     00007ff9de5351a0
Size:        64(0x40) bytes
File:        C:\Program Files\dotnet\shared\Microsoft.NETCore.App\2.1.6\System.Private.CoreLib.dll
Fields:
              MT    Field   Offset                 Type VT     Attr            Value Name
00007ff9dee80850  4000295        8        System.Object  0 instance 0000012951807dc0 _target
00007ff9dee80850  4000296       10        System.Object  0 instance 0000000000000000 _methodBase
00007ff9dee971f8  4000297       18        System.IntPtr  1 instance     7ff98a574090 _methodPtr
00007ff9dee971f8  4000298       20        System.IntPtr  1 instance     7ff9de5873c0 _methodPtrAux
00007ff9dee80850  40002a2       28        System.Object  0 instance 00000129516e2748 _invocationList
00007ff9dee971f8  40002a3       30        System.IntPtr  1 instance              815 _invocationCount
0:000> !do 00000129516e2748
Name:        System.Object[]
MethodTable: 00007ff9dee63878
EEClass:     00007ff9de544600
Size:        32792(0x8018) bytes
Array:       Rank 1, Number of elements 4096, Type CLASS
Fields:
None
levi106 commented 5 years ago

If failed to connect to the IoT Hub, CloudConnectionProvider.Connect method returns error.

2019-06-07 14:50:44.690 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Error creating cloud connection for client myEdgeDevice/$edgeHub
2019-06-07 14:51:05.730 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Error creating cloud connection for client myEdgeDevice/CSharpModule
2019-06-07 14:52:26.774 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Error creating cloud connection for client myEdgeDevice/CSharpModule
2019-06-07 14:52:47.801 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Error creating cloud connection for client myEdgeDevice/$edgeHub
2019-06-07 14:53:08.829 +00:00 [WRN] [Microsoft.Azure.Devices.Edge.Hub.CloudProxy.CloudConnection] - Error creating cloud connection for client myEdgeDevice/CSharpModule

Therefore, ConnectionManager.GetOrCreateCloudConnection will call CloudConnectionProvider.Connect method every time. However, ConnectivityAwareClient.HandleDeviceConnectedEvent will not be removed from DeviceConnectivityManager.DeviceConnected, ConnectivityAwareClient objects and its associated objects which are created by CloudConnectionProvider will continue to leak.

varunpuranik commented 5 years ago

@levi106 - Thanks for reporting this. I will look into this asap.

levi106 commented 5 years ago

@varunpuranik Is there any update?

varunpuranik commented 5 years ago

@levi106 - The leaking of event handlers has been fixed in master - https://github.com/Azure/iotedge/commit/4f052756228bf84ac663e4f6171a0a47aa2025ae#diff-98a736262e0091d4cc8c5192d8ce48a8 It will be part of the 1.0.9 release.

levi106 commented 5 years ago

I see. Thanks.

github-actions[bot] commented 4 years ago

This issue is being marked as stale because it has been open for 30 days with no activity.

lt72 commented 4 years ago

Closing this issue as it seems resolved. Please re-open as needed.