[BUG] With datadog/agent:7.55.3-jmx I cannot exclude java.lang metrics

[aonamrata]

I am trying to configure my kafka debezium connector with JMX metrics and found that we are not sending all the metrics. Some of them are lost because they are string or map etc which is fine. But some are truncated because there are too many metrics. This is the message in logs.

2024-08-07 16:00:29 2024-08-07 10:30:29 UTC | JMX | INFO | Instance | Found 356 matching attributes
2024-08-07 16:00:29 2024-08-07 10:30:29 UTC | JMX | WARN | CustomLogger | Number of returned metrics is too high for instance: jmx_instance. Please read http://docs.datadoghq.com/integrations/java/ or get in touch with Datadog Support for more details. Truncating to 350 metrics.

Also in datadog website > Metrics summary > filter by service name -- i only get 266 metrics. So we definitely are not over the limit.

Agent Environment FROM datadog/agent:7.55.3-jmx

Config:

init_config:
    is_jmx: true
    collect_default_metrics: true
    new_gc_metrics: true
    service: {{ SERVICE_NAME|default("kafka_connect") }}
    conf:
      - include:
          domain: 'debezium.mysql'
      - exclude:
          attribute: 
            - LastAppliedChange
            - LastRecoveredChange
            - Status
            - LastEvent
            - ChunkId
            - ChunkFrom
            - ChunkTo
            - TableFrom
            - TableTo
            - GtidSet
            - BinlogFilename
            - LastEvent
            - SourceEventPosition
            - LastTransactionId
            - Version
            - CommitId
            - BufferPool
            - Name
            - Type
            - UsageThreshold
            - commit-id
            - version

logs: []

instances:
  - host: localhost
    port: 9095
    rmi_registry_ssl: false
    name: jmx_instance

Describe what you expected:

1. Shouldn't i get most of the metrics in datadog website.
2. If we are exceeding the limit cant I exclude the java.lang and other java.* metrics? We get some of them from datadog integration with AWS ECS.

Steps to reproduce the issue: This is the Dockerfile content

FROM datadog/agent:7.55.3-jmx

LABEL dd.agent.version="7.55.3"

RUN apt-get -y update \
  && apt-get -y upgrade \
  && apt-get -y clean \
  && rm -rf /var/lib/apt/lists/*

COPY conf.yaml /etc/datadog-agent/conf.d/kafka.d/conf.yaml

[vickenty]

Hi, thank you for using Datadog.

The number on the metric summary page is lower because it shows number of distinct metric names, not taking tags into account (total number is visible after clicking on the metric name). Each matching JMX attribute becomes a metric, but multiple beans can use the same metric name, with bean attributes becoming metric tags (for an example please see the default config).

To exclude java metrics, you can add this filter to your JMX integration config:

- exclude:
  domain: java.lang

If you have any further questions, please feel free to contact Datadog support.