search

DataDog / dd-agent

Datadog Agent Version 5
https://docs.datadoghq.com/
Other
1.3k stars 812 forks source link

Docker: Datadog agent stops sending default JMX metrics after adding bean #3902

Open yoldar opened 3 years ago

yoldar commented 3 years ago

Datadog docker agent version: datadog/agent:latest-jmx

Configuration before adding bean conf.d/jmx.d/conf.yaml:

init_config:
  is_jmx: true
  collect_default_metrics: true
  new_gc_metrics: true

instances:
  - host: register
    port: 5000

Configuration above works fine. But after adding bean below:

init_config:
  is_jmx: true
  collect_default_metrics: true
  new_gc_metrics: true

instances:
  - host: register
    port: 5000
    conf:
      - include:
        domain: com.zaxxer.hikari:type=Pool (HikariPool-1)
        bean:
          - com.zaxxer.hikari
        attribute:
          IdleConnections:
            metric_type: gauge
            alias: jmx.com.zaxxer.hikari.idle_connections
          ActiveConnections:
            metric_type: gauge
            alias: jmx.com.zaxxer.hikari.active_connections
          TotalConnections:
            metric_type: gauge
            alias: jmx.com.zaxxer.hikari.total_connections
          ThreadsAwaitingConnection:
            metric_type: gauge
            alias: jmx.com.zaxxer.hikari.threads_awaiting_connection

Default JMX metrics are not sent because of this logs:

 INFO | (pkg/jmxfetch/jmxfetch.go:304 in func1) | 2020-11-21 18:24:56 UTC | JMX | WARN  | JmxAttribute | Unable to get metrics from JMImplementation:type=MBeanServerDelegate - MBeanServerId: java.lang.NumberFormatException: For input string: "9b6c307b23cf_1605982746024"
 INFO | (pkg/jmxfetch/jmxfetch.go:304 in func1) | 2020-11-21 18:24:56 UTC | JMX | WARN  | JmxAttribute | Unable to get metrics from JMImplementation:type=MBeanServerDelegate - SpecificationName: java.lang.NumberFormatException: For input string: "Java Management Extensions"
 INFO | (pkg/jmxfetch/jmxfetch.go:304 in func1) | 2020-11-21 18:24:56 UTC | JMX | WARN  | JmxAttribute | Unable to get metrics from JMImplementation:type=MBeanServerDelegate - SpecificationVendor: java.lang.NumberFormatException: For input string: "Oracle Corporation"
 INFO | (pkg/jmxfetch/jmxfetch.go:304 in func1) | 2020-11-21 18:24:56 UTC | JMX | WARN  | JmxAttribute | Unable to get metrics from JMImplementation:type=MBeanServerDelegate - ImplementationName: java.lang.NumberFormatException: For input string: "JMX"
 INFO | (pkg/jmxfetch/jmxfetch.go:304 in func1) | 2020-11-21 18:24:56 UTC | JMX | WARN  | JmxAttribute | Unable to get metrics from JMImplementation:type=MBeanServerDelegate - ImplementationVersion: java.lang.NumberFormatException: multiple points
 INFO | (pkg/jmxfetch/jmxfetch.go:304 in func1) | 2020-11-21 18:24:56 UTC | JMX | WARN  | JmxAttribute | Unable to get metrics from JMImplementation:type=MBeanServerDelegate - ImplementationVendor: java.lang.NumberFormatException: For input string: "Oracle Corporation"
 INFO | (pkg/jmxfetch/jmxfetch.go:304 in func1) | 2020-11-21 18:24:56 UTC | JMX | WARN  | JmxAttribute | Unable to get metrics from java.lang:type=Runtime - SpecName: java.lang.NumberFormatException: For input string: "Java Virtual Machine Specification"

If I remove bean and restart agent default metrics start to send. What is wrong?

ogaca-dd commented 3 years ago

@yoldar,

Unable to get metrics from is just a warning meaning that JMXFetch does not support directly the type String and you can safely ignore these warnings.

Can you try running datadog-agent jmx list collected and datadog-agent jmx list everything? There are also other debugging commands for JMX that can help understanding your issue.

List attributes matched by JMXFetch.

Usage:
  datadog-agent jmx list [command]

Available Commands:
  collected         List attributes that will actually be collected by your current instances configuration.
  everything        List every attributes available that has a type supported by JMXFetch.
  limited           List attributes that do match one of your instances configuration but that are not being collected because it would exceed the number of metrics that can be collected.
  matching          List attributes that match at least one of your instances configuration.
  not-matching      List attributes that don’t match any of your instances configuration.
  with-metrics      List attributes and metrics data that match at least one of your instances configuration.
  with-rate-metrics List attributes and metrics data that match at least one of your instances configuration, including rates and counters.
skysbsb commented 9 months ago

@yoldar did u manage to fix it? how?