DataDog / java-dogstatsd-client

Java statsd client library
MIT License
176 stars 101 forks source link

StatsD Client Keeps Creating Threads and Never Releasing Them #254

Closed canpan14 closed 3 months ago

canpan14 commented 3 months ago

I have a Java 17/Spring Boot 3 project and did a basic StatsD client bean setup to emit metrics to Datadog when my route is called. The issue is seems to keep spinning up more StatsD-Processor-1 and StatsD-Sender-1 threads with every call and never releasing them. The metrics are reporting fine in Datadog so it's not like they aren't making it there.

Local profiling:

image

In a deployed environment where we ran perf tests but I didn't notice the issue for days. You can see it seems to never release these threads.

image

From a small window in the Datadog Profiler image

Below is a trimmed down version of all the code is doing.

@Component
public class RequestFilter implements HandlerInterceptor {
    private final DatadogStatsDClient datadogClient;

    public RequestFilter(DatadogStatsDClient datadogClient) {
        this.datadogClient = datadogClient;
    }

    @Override
    public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws IOException {
        this.datadogClient.sendInvokeMetric();
    }
}
@Component
public class DatadogStatsDClient {
    private final StatsDClient statsDClient;

    public DatadogStatsDClient() {
        this.statsDClient = new NonBlockingStatsDClientBuilder()
                .hostname("localhost")
                .port(8125)
                .build();
    }

    public void sendInvokeMetric() {
        this.statsDClient.count("myroute.invocations", 1);
    }
}

I'm not quite sure what is going on as the setup seems pretty simple.