Spring Boot 3.x Micrometer StatsD registry does not send metrics to UDP port when another StatsD client is active on the same port

[spachari-im]

Describe the bug In Spring Boot 3.x, Micrometer's inbuilt StatsD agent does not report JMX metrics when another StatsDClient is created to send custom metrics to the same host & port.

Environment Spring Boot 3.2.7, Spring Camel version 4.5.0 application.

• Micrometer version [1.12.7]
• Micrometer registry - StatsD [1.9.2]
• OS: MacOS
• Java version: 17

To Reproduce How to reproduce the bug:

• Declare a StatsDClient bean to send custom metrics to Datadog

  StatsDClient statsDClient = new NonBlockingStatsDClientBuilder()
                  .prefix(PREFIX)
                  .hostname(localhost)
                  .port(8125)
                  .build(); 

• Add mcrometer-registry-statsd to project (in pom.xml).

• Enable micrometer statsd metrics.

  management.statsd.metrics.export.enabled=true
  management.statsd.metrics.export.flavor=datadog
  management.statsd.metrics.export.protocol=udp
  management.statsd.metrics.export.host=localhost
  management.statsd.metrics.export.port=8125
  management.statsd.metrics.export.polling-frequency=10s
  management.statsd.metrics.export.publish-unchanged-meters=true

  Note: the Micrometer StatsD client and the custom StatsDClient both send metrics to the same host and port

• The application does not report the implicit JVM metrics sent by Micrometer to the StatsD Agent to port 8125 (netcat -ulp 8125).

• Only the custom metrics sent by the Datadog StatsD client is being reported to port 8125.

• When the Datadog StatsD client is removed, the JVM metrics are successfully reported to port 8125.

Expected behavior

• On port 8125 (netcat -ulp 8125) both the custom metrics sent through Datadog StatsD client and the internal JVM metrics sent by Micormeter should be visible.
• Micrometer should report some warning in logs if metrics cannot be sent to the UDP port, instead of ignoring the metric.

Additional context

• This configuration runs successfully on Spring Boot 2.x and Micrometer 1.9.2.
• When the Datadog StatsD client is removed, the JVM metrics are successfully reported to port 8125.
• It appears that the Micrometer StatsD agent does not work if there is another StatsD client sending to the same port (I am not sure of this)
• There are no warnings in the logs to give an idea of why this happens.

[shakuzen]

Thank you for the report, and sorry for the late triage on it.

• Micrometer version [1.12.7]
• Micrometer registry - StatsD [1.9.2]

You should use a single, consistent version of Micrometer. If you are using Spring Boot's dependency management, it should manage the version for you so you don't end up with two different versions like this. Using different versions has potential to cause issues.

I tried to reproduce the issue and I was not able to do so. I added the Dogstatsd client (StatsDClient) to a Spring Boot 3.4.0 app with Micrometer's statsd registry, and all metrics (from each client) were available in Datadog. Could you please try with the latest versions and ensure consistent versions are used to see if this is still an issue for you?

[spachari-im]

@shakuzen thanks for triaging this.

I did not override the micrometer version. I just specified it in the ticket. The only micrometer dependency in the propject's pom file is this:

<dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-statsd</artifactId>
    </dependency>

I tried this test after upgrading the Spring Boot version to 3.4.0. I get the same behavior - if a StatsD client is added for custom metrics, the custom metrics are reported but JVM metrics are not shown.

The JVM metrics are shown when the StatsD client is removed.

I think this is because of a setting enabled in the properties file or a dependency related to the Camel starter dependency - org.apache.camel.springboot.

Can you please post all the code you have used (properties enabled, bean declaration and pom dependency)?

[shakuzen]

Could you start with a minimal project and verify first whether you see the issue there, and then add things like the Camel starter to see if it affects things? The project I used is a bit complicated, but I expect it would work the same in a minimal project, so we need to figure out what in your project is causing the difference.