Application Insights CPU metrics is not valid for Azure Spring Apps

[felipmiguel]

Describe the bug Application Insights CPU metric is not properly measured in Azure Spring Apps. In Azure Spring Apps it is possible to have application with different vCPU assigned, up to 4 vCPU. However the underlaying virtual machines seems to have 16 cores.

When Application Insights calculates de CPU utilization, for instance in the Live Metrics tab, the maximum CPU utilization it shows corresponds to number of vCPU configured in Azure Spring Apps divided by the host physical CPU. Eg, if the app has 4vCPU and VM has 16cores the max CPU shown is 4/16 -> 1/4 -> 25%.

If the app has 1vCPU -> 1/16-> 6,25%. This scenario was detected with Java applications, not verified with other runtimes.

On the app logs detected the following JVM setting injected by Azure Spring Apps: JAVA_OPTIONS has -XX:ActiveProcessorCount=16

To Reproduce Steps to reproduce the behavior:

1. Ensure Azure Spring Apps has Application insights enabled
2. Create an application in Azure Spring Apps with 1CPU.
3. Deploy the application (Java).
4. Execute a load test against the deployed application.
5. Looking at Java metrics verify that process.cpu.usage/system.cpu.usage is 1 or near 1.
6. Open Application Insights and Live Metrics. Look for the application. The CPU Total is 1/16 -> 6,25%. This value is not correct, it should show 100%.

Expected behavior The CPU total in Application Insights should be 100 x process.cpu.usage. In above case 100%.

Screenshots

Additional context On the app logs detected the following JVM setting injected by Azure Spring Apps: JAVA_OPTIONS has -XX:ActiveProcessorCount=16

Can we contact you for additional details? Y

If yes, please send us your contact information to AzureSpringCloud-Talk@service.microsoft.com and include the issue number in the email title.

[LGDoor]

Is it possible to switch to the metrics from Azure Monitor? App CPU Usage is the most accurate one.

[felipmiguel]

Thanks @LGDoor I'm using process.cpu.usage and I see that results are aligned with App CPU Usuage. However, the issue that I see is that CPU Total in Application Insights live metric is not real. In the latest test I'm performing I see it is max 50%. I checked the logs of the application and I see again Setting Active Processor Count to 8:

Setting Active Processor Count to 8
Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=209353K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 8G, Thread Count: 250, Loaded Class Count: 34548, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 127 container CA certificates to JVM truststore
Configuring Azure Application Insight properties
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_microsoft-openjdk/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -javaagent:/layers/paketo-buildpacks_azure-application-insights/azure-application-insights-java/applicationinsights-agent-3.4.2.jar -XX:ActiveProcessorCount=8 -Deureka.client.service-url.defaultZone=https://fitness-store-prod-vnet.svc.private.azuremicroservices.io/eureka/default/eureka -Xms5800m -Xmx5800m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.local.only=true -Dmanagement.endpoints.jmx.exposure.include=health,metrics -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dspring.jmx.enabled=true -Dserver.tomcat.mbeanregistry.enabled=true -Dfile.encoding=UTF8  -XX:MaxDirectMemorySize=10M -XX:MaxMetaspaceSize=209353K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics
OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
2023-02-22 11:54:43.001Z INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.4.2 started successfully (PID 1, JVM running for 3.175 s)
2023-02-22 11:54:43.006Z INFO  c.m.applicationinsights.agent - Java version: 17.0.4.1, vendor: Microsoft, home: /layers/paketo-buildpacks_microsoft-openjdk/jdk

As the application has 4CPU assigned, what I see in Application insights is 4CPU assigned/8 Active Processor Count = 0.5 -> 50%

The integration with Application Insights doesn't work as expected. Same application running on Azure App Services or Azure Container Apps is properly shown in App Insights.

The issue seems related to that Setting Active Processor Count to 8. A couple of months ago I performed same test there was _JAVAOPTIONS has -XX:ActiveProcessorCount=16 in the logs, and CPU in app insights was 4CPU assigned/16 Active Processor Count = 0.25 -> 25%

[domainname]

@felipmiguel I'm assuming that you are using Enterprise tier. It appears that this is related to this buildpacks issue. Waiting for buildpacks to provider a way to stop injecting -XX:ActiveProcessorCount. In that issue, a workaround is mentioned, which is to manually pass -XX:ActiveProcessorCount= to override the one that buildpacks injects.

[felipmiguel]

Hi @domainname. I'm using Enterprise tier. The behavior will be different if I deploy a jar file instead of using a build?

[seanli1988]

Jar still uses buildpacks in Enterprise but it may not trigger the same buildpack issue. worth to give it a try.

[domainname]

In https://github.com/paketo-buildpacks/libjvm/issues/136#issuecomment-1027973325, a workaround is mentioned. You may manually set -XX:ActiveProcessorCount=<actual-cpu-count> like this in ASA.

[felipmiguel]

Thanks @domainname. It works appliying that setting.