Application Insights runtime attach agent metric-filtering not working

tamasvincze commented 4 months ago

Hi all,

I'm using com.microsoft.azure:applicationinsights-runtime-attach:3.4.19 and com.microsoft.azure:applicationinsights-core:3.4.19 dependecies for my spring boot 3.2.2 application with Azure App Insight. I'm trying to exclude certain metrics for my application, without any success.

Expected behavior

The excluded metrics should be sent to azure app insight

Actual behavior

The metrics are still sent to app insight.

System information

Please provide the following information:

SDK Version: 3.4.19
OS type and version: Windows 11
Application Server type and version (if applicable): standalone java app
Using spring-boot? Yes, version 3.2.2
Additional relevant libraries (with version, if applicable): -

applicationinsights.json content:

{
  "sampling": {
    "requestsPerSecond": 0.5
  },
  "preview": {
    "sampling": {
      "requestsPerSecond": 0.5,
      "overrides": [
        {
          "telemetryType": "dependency",
          "percentage": 0
        }
      ]
    },
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  }
}

Screenshots

Screenshot showing excluded metrics:

heyams commented 4 months ago

@tamasvincze your config looks good to me. it didn't work probably because config didn't get picked up. Since you enabled App Insights Java programmatically, the configuration file will be read from the classpath ((src/main/resources, src/test/resources). or use system property applicationinsights.runtime-attach.configuration.classpath.file to specify the json config file path.

https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-spring-boot#configuration-file-location

heyams commented 4 months ago

@tamasvincze we GA sampling overrides since 3.5.0 and the latest is 3.5.1 GA.

tamasvincze commented 4 months ago

Hey @heyams, thanks for the responses. My applicationinsights.json file is on the classpath: C:\Dev\redacted-app\src\main\resources\applicationinsights.json

I upgraded to 3.5.1:

<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>applicationinsights-core</artifactId>
    <version>3.5.1</version>
</dependency>
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>applicationinsights-runtime-attach</artifactId>
    <version>3.5.1</version>
</dependency>

My current config file (on the same location:

{
  "sampling": {
    "requestsPerSecond": 0.5
  },
  "preview": {
    "sampling": {
      "requestsPerSecond": 0.5,
      "overrides": [
        {
          "telemetryType": "dependency",
          "percentage": 0
        }
      ]
    }
  },
  "processors": [
    {
      "type": "metric-filter",
      "exclude": {
        "matchType": "strict",
        "metricNames": [
          "process_cpu_usage",
          "log4j2_events",
          "jvm_threads_states",
          "executor_pool_core",
          "executor_pool_max",
          "jvm_gc_overhead",
          "% Of Max Heap Memory Used",
          "Loaded Class Count"
        ]
      }
    }
  ]
}

In the log I see the newest version being used: 2024-03-13 17:46:52.829Z INFO c.m.applicationinsights.agent - Application Insights Java Agent 3.5.1 started successfully (PID 32080, JVM running for 5.752 s)

However on the azure portal, I still see the event that should have been excluded:

heyams commented 4 months ago

please try this instead without preview for sampling:

{
  "sampling": {
      "requestsPerSecond": 0.5,
      "overrides": [
        {
          "telemetryType": "dependency",
          "percentage": 0
        }
      ]
   }, 
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  }
}

if you set the self-diagnostics log to debug level, it should output your json config. that is one way to verify that your config gets picked up.

tamasvincze commented 4 months ago

I'm added selfDiagnostics to the json:

{
  "sampling": {
    "requestsPerSecond": 0.5,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 0
      }
    ]
  },
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  },
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "DEBUG",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  }
}

The applicationinsights.log in the root of my project does not show any debug level info:

2024-03-13 18:09:31.441Z WARN  c.a.c.h.netty.implementation.Utility - The following Netty dependencies have versions that do not match the versions specified in the azure-core-http-netty pom.xml file. This may result in unexpected behavior. If your application runs without issue this message can be ignored, otherwise please update the Netty dependencies to match the versions specified in the pom.xml file. Versions found in runtime: 'io.netty:netty-common' version not found (expected: 4.1.101.Final),'io.netty:netty-handler' version not found (expected: 4.1.101.Final),'io.netty:netty-handler-proxy' version not found (expected: 4.1.101.Final),'io.netty:netty-buffer' version not found (expected: 4.1.101.Final),'io.netty:netty-codec' version not found (expected: 4.1.101.Final),'io.netty:netty-codec-http' version not found (expected: 4.1.101.Final),'io.netty:netty-codec-http2' version not found (expected: 4.1.101.Final)
2024-03-13 18:09:33.742Z INFO  c.m.applicationinsights.agent - Application Insights Java Agent 3.5.1 started successfully (PID 20072, JVM running for 7.776 s)
2024-03-13 18:09:33.745Z INFO  c.m.applicationinsights.agent - Java version: 17.0.10, vendor: Oracle Corporation, home: C:\Program Files\Java\jdk-17.0.10

However my application is logging out all properties through Spring's ConfigurableEnvironment, and I see this printed:

Mar 13 2024 18:09:48 main INFO  redacted.app.PropertySourceLogger [] - applicationinsights.internal.runtime.attached.json={  "sampling": {    "requestsPerSecond": 0.5,    "overrides": [      {        "telemetryType": "dependency",        "percentage": 0      }    ]  },  "preview": {    "processors": [      {        "type": "metric-filter",        "exclude": {          "matchType": "strict",          "metricNames": [            "process_cpu_usage",            "log4j2_events",            "jvm_threads_states",            "executor_pool_core",            "executor_pool_max",            "jvm_gc_overhead",            "% Of Max Heap Memory Used",            "Loaded Class Count"          ]        }      }    ]  },  "selfDiagnostics": {    "destination": "file+console",    "level": "DEBUG",    "file": {      "path": "applicationinsights.log",      "maxSizeMb": 5,      "maxHistory": 1    }  }}

Sadly, I still see the metrics being sent to azure:

heyams commented 4 months ago

@tamasvincze i will try it myself and let you know.

heyams commented 4 months ago

@tamasvincze Here is my sample for metric-filter. it works for me. i used the latest 3.5.1 GA.

We reserve certain default metrics not to be filtered out intentionally. You should be able to filter out any other custom metrics though.

In my applicationinsights.log, it showed clearly that json config was applied successfully: DEBUG c.m.a.a.i.c.ConfigurationBuilder - configuration: {"connectionString":

tamasvincze commented 4 months ago

Your configuration made me try something.

Here is my current config:

{
  "sampling": {
    "requestsPerSecond": 0.5,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 0
      }
    ]
  },
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "DEBUG",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  },
  "instrumentation": {
    "logging": {
      "enabled": "true"
    }
  },
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  }
}

And for my app, I have 2 environmental variables exported (in the IDEA runconfig):

APPLICATIONINSIGHTS_ROLE_NAME=test-vinctam12
APPLICATIONINSIGHTS_CONFIGURATION_CONTENT={"connectionString":"InstrumentationKey=REDACTED;IngestionEndpoint=https://northeurope-0.in.applicationinsights.azure.com/;LiveEndpoint=https://northeurope.livediagnostics.monitor.azure.com/"}

I don't see any DEBUG logs from the agent.

On Azure Portal I see the excluded metrics:

However if I delete the env vars and use role configuration through the json, like this:

{
  "connectionString": "InstrumentationKey=REDACTED;IngestionEndpoint=https://northeurope-0.in.applicationinsights.azure.com/;LiveEndpoint=https://northeurope.livediagnostics.monitor.azure.com/",
  "role": {
    "name": "test-vinctam13",
    "instance": "instance1"
  },
  "sampling": {
    "requestsPerSecond": 0.5,
    "overrides": [
      {
        "telemetryType": "dependency",
        "percentage": 0
      }
    ]
  },
  "selfDiagnostics": {
    "destination": "file+console",
    "level": "DEBUG",
    "file": {
      "path": "applicationinsights.log",
      "maxSizeMb": 5,
      "maxHistory": 1
    }
  },
  "instrumentation": {
    "logging": {
      "enabled": "true"
    }
  },
  "preview": {
    "processors": [
      {
        "type": "metric-filter",
        "exclude": {
          "matchType": "strict",
          "metricNames": [
            "process_cpu_usage",
            "log4j2_events",
            "jvm_threads_states",
            "executor_pool_core",
            "executor_pool_max",
            "jvm_gc_overhead",
            "% Of Max Heap Memory Used",
            "Loaded Class Count"
          ]
        }
      }
    ]
  }
}

Now I see DEBUG logs, like:

2024-03-13 20:33:49.724Z DEBUG c.m.a.a.i.c.ConfigurationBuilder - read system property: applicationinsights.internal.runtime.attached.json=...
2024-03-13 20:33:49.728Z DEBUG c.m.a.a.i.c.ConfigurationBuilder - configuration: {"connectionString":"InstrumentationKey=...

And now I only see one of the excluded metrics:

Can you try exporting the rolename and connection string instead of having them in the json?

heyams commented 4 months ago

@tamasvincze APPLICATIONINSIGHTS_CONFIGURATION_CONTENT has higher precedence over the json config. that is why when this is present and you only set connection string in it, nothing gets applied.

APPLICATIONINSIGHTS_CONFIGURATION_CONTENT is equivalent to the json config. For connection string, you can use this env var called APPLICATIONINSIGHTS_CONNECTION_STRING or specify it in the json config.

It's great to hear that it's working for you. "Loaded Class Count" can't be filtered out. Please let me know if i can close this issue.

tamasvincze commented 4 months ago

Oh you're right, I've been using the wrong env var. Thanks for the help! I'll close the issue :)

microsoft / ApplicationInsights-Java