search

microsoft / ApplicationInsights-Profiler-AspNetCore

Application Insights Profiler sample and documentation
MIT License
66 stars 24 forks source link

Application Insights Profiler causing high memory usage #188

Closed vigneshmsft closed 1 year ago

vigneshmsft commented 1 year ago

I have an AspNetCore (.NET 6) application running on Azure App Services and decided to enable application insights and profiler in our production environment this week as we were preparing for a larger rollout of our application. The application is packed as a docker container and runs as a linux container web app in App Services. App Services plan is P2V2.

I noticed the increase in CPU and memory usage right after the change and disabled random sampling from application insights profiler interface in Azure portal and only had it to profile if it exceeded a CPU or memory threshold. I understand profiling can increase CPU and memory usage but the average memory usage was 10x the usual average for my application and also when the profiler was not running. There was a noticeable increase in the CPU% for the app service too.

Memory usage after enabling AddServiceProfiler() using default configuration. image The drop in memory usage back to the normal ~150MB is when I removed the AppServiceProfiler() config in code and redeployed.

Environment: Azure App Service P2V2 AspNetCore 6.0.11 running as docker container

nuget package versions: Microsoft.ApplicationInsights.AspNetCore (2.21.0) Microsoft.ApplicationInsights.Profiler.AspNetCore (2.5.0)

xiaomi7732 commented 1 year ago

Hi @vigneshmsft Our memory sampling shows a level of usage increasing but not 10x as much:

image (WorkingSet in MB)

When you evaluate the memory usage, was it the usage for the whole app service plan? How many application instances were there on the plan? Were they running profiling at the same time?

vigneshmsft commented 1 year ago

there is only one app on the plan, it was running 2 instances at the time, I also have a deployment lot for the production instance.

vigneshmsft commented 1 year ago

the attached image was the usage metric for the app and not the app service as whole.

xiaomi7732 commented 1 year ago

Hi @vigneshmsft, I deployed a containerized .NET 6 webapi to a P2V2 plan for testing the profiler, here's my results:

image

In the last 12 hours, there's been up and down. When it is up, it's profiling. And then, it went down. Ranging from 180MB to 220MB. I still can't see what was described. Am I looking into the same metrics as you do?

To further troubleshoot, we need to understand what is taking the memory. Is it possible for you to take 2 memory dumps, one before the high usage, and the other during the profiler, when the memory usage is higher, so that we can compare the dumps to see what actually is using the memory?

vigneshmsft commented 1 year ago

sorry it is not possible for me to do this now at this point for this application, we have refrained from using the profiler.

xiaomi7732 commented 1 year ago

Hi Vigneshmsft, okay. Thanks for the update. Let us know if you decide to give it another shoot.