search

statsd / statsd

Daemon for easy but powerful stats aggregation
MIT License
17.61k stars 1.96k forks source link

StatsD repeater backends cause data lose #617

Open rrfeng opened 7 years ago

rrfeng commented 7 years ago

We have a statsd run on UDP and InfluxDB backend worked fine, until the day before yesterday when we add a repeater backend to the statsd instance. Repeater send data to another statsd and there is a backend write by ourselves, for some extra analysis. At first the two statsd worked fine, but at 20:00 yesterday, the problem came. We use grafana with InfluxDB backend, and the counter shows only 50/s which should be aoubt 100/s (tps of an api). And same at tonight 20:00. (traffic going heavy at the time.) I remove the repeater backend and restart the statsd, all things going to be ok. How should I resolve this?

elifish4 commented 7 years ago

Hi@rrfeng , I have similar problem. Did you managed to solve this issue?

rrfeng commented 7 years ago

@elifish4 yes, I found out that was because the cpu load. As you know, nodejs only use single core, when the cpu usage being 100%, it start losing data.

I made a trick: before: 

source -> statsd(repeater and store) ->[( influxdb )(statsd -> xxxxx)]

after:

source -> statsd(repeater only) -> [(statsd -> influxdb)(statsd -> xxxx)]