DogStatsD starts increasing metrics indefinitely if it fails to flush metrics

[halkar]

Recently we faced a problem with our DataDog metrics. We have several metrics we are sending from our ASP.Net Core app, and most of them are simple counter metrics. We call the IDogStatsd.Increment method. 99.999% of all metrics are OK and sent to DataDog without issues. But sometimes we generate a metric that doesn't fit the buffer, and then we get an InvalidOperationException. See code here https://github.com/DataDog/dogstatsd-csharp-client/blob/02d6a27cb6458f44600564114b85a3edb5678aed/src/StatsdClient/Bufferize/BufferBuilder.cs#L62 It's not a problem by itself, and we've already fixed the issue on our side. But other counters using IDogStatsd.Increment go nuts when it happens (all counter metrics are constantly increasing).

Looks like DogStatsD keeps trying to flush the metrics but keeps crashing on the broken one. The only way to solve this problem is to kill the service. I guess the broken metric should be excluded rather than retried every time. Looks like the problem started in version 7.0.0. Also, as the flush process runs asynchronously, we cannot handle this exception, making it hard to investigate the issue. DogStatsD doesn't return or write the exception to logs.

[ogaca-dd]

@halkar, Thank you for reporting this issue. I will try to reproduce the issue and create a new release that fix the problem.

[ogaca-dd]

#178 should fix this issue. I am going to create a new release at the beginning of next week.

[halkar]

Awesome! Thanks @ogaca-dd

[ogaca-dd]

@halkar Thank you again for reporting this issue. DogStatsD-CSharp-Client 7.0.1 should fix your issue. Let us know if it is not the case.

[halkar]

Thanks a lot @ogaca-dd