[V2] Memory leak for Edge Hub and Edge Agent

[helenhllo]

I ran the quick start tutorial on my Ubuntu vm On my Vm I have the following docker containers running: edgeAgent, edgeHub, & tempSensor

Azure VM info:

Operating System:  Ubuntu 16.04.3 LTS (GNU/Linux 4.13.0-1011-azure x86_64)
OSType: Linux
Size: Standard B1ms (1 vcpu, 2 GB memory)
OS Disk: 30GB

Edge Container info:

microsoft/azureiotedge-hub: 
- "Created":"2018-03-02T03:42:47.081358486Z"
- "imageHash": "sha256:42f176e6d2b5df3206c8188d0455d4e55a140dceccd040aedad028bea1b2b431"
- Running 3 days

microsoft/azureiotedge-agent:
- "Created": "2018-03-02T03:41:12.272644595Z"
- "imageHash": "sha256:a604f2d8d15f15d9f2b75049d8721e596975babdb00cb0ee81bd4ae351625b53"
- Running 3 days

microsoft/azureiotedge-simulated-temperature-sensor:  
- "Created": "2018-03-02T03:43:24.612608681Z"
- "imageHash": "sha256:e33fcbbfa1b0d75d4faf5094fc8403c97812691c6eb4c5334222ba098dc38eb2"
- Running 3 days

I've used smem tool to monitor the memory used of each process related to Iot Edge. Exported the data to .csv and create a graph: As you can see from the graph, both Edge Hub and Edge Agent are gradually using up more memories. Free is the amount of free memory on the system.

Please let me know if I need to provide any other detail/logs. Thanks

[Jan-Olof]

Interesting. Could this cause the failures of the iot-edge-opc-publisher that we have experienced I wonder?

https://github.com/Azure/iot-edge-opc-publisher/issues/110

[varunpuranik]

Thanks for the analysis and information. We have not noticed memory leak issues from the IoT Edge Runtime before. We will investigate this issue.

[dariuszparys]

I experienced similar memory usage in a custom dotnet core module it was eating up to 500 MB. I limited the available memory for the container to 50MB and it just consumed from this point of time 25MB. So I'm not sure if this is really a memory leak.

Can you try this again with memory limits set?

[helenhllo]

@dariuszparys Isn't the simulator container (microsoft/azureiotedge-simulated-temperature-sensor) also a dotnet core module? but it's memory doesn't increase as the server runs. Doesn't that negate your assumption?

I'll try your way and let you know

[dariuszparys]

@helenlo319 you're right just had a look at the chart you posted, I assume that routes have been configured to $upstream?

[ts4iot]

@dariuszparys, can you please explain how to limit the available memory for the edgeHub and edgeAgent modules. I try it but it doesn't work. I assume it should be done in:

SetModules->Advanced Edge Settings->Create Options

Can you please provide a JSON example that works for edgeHub and edgeAgent.

Thanks

[dariuszparys]

@ts4iot basically I just did that during public preview with custom modules. The createOptions of the edgeAgent and edgeHub should also work in my opinion.

To specify a limit of 50MB on your running image you have to configure createOptions with

{
  "Hostconfig": {
    "Memory": 52428800
  }
}

But haven't tested it on GA bits

[BewaAutomatisierung-RD]

Limiting the memory for EdgeHub is to no avail. It just causes the EdgeHubModule to be "killed" once it surpasses the memory limit. In my opinion this is not much better than an EdgeHub Module crashing because the system has run out of memory. One exception though: At least you know that the EdgeHubModule is killed and not something else.

If you want to limit memory on a Raspberry Pi, be aware that memory limit support is turned off by default. You can find a HowTo for enabling it here: https://blog.raveland.org/post/docker_raspian/

[varunpuranik]

@BewaControl-ReneDivossen - Did you try to limit the memory with only MQTT protocol head enabled? We have tested that, and it should work. Please let us know if it doesn't work for you and we will look into it.

[BewaAutomatisierung-RD]

@varunpuranik - yes I did try that on a virtual machine on azure. The result was as described above - edgeHub was killed when surpassing the limit. Unfortunately limiting memory for the EdgeHub it did not cause EdgeHub to free memory when approaching the limit.