search

uber-archive / statsrelay

A consistent-hashing relay for statsd and carbon metrics
Other
101 stars 28 forks source link

Error from send: Message too long #71

Open keyboardfann opened 7 years ago

keyboardfann commented 7 years ago

Hello All, I start statsrelay instances and use statsd-tg to do performance testing. After push metrics minutes, statsrelay show error "Error from send: Message too long". Is it because of the queue full and drop message?

Test script 

while true;do statsd-tg -d 10.62.4.240 -D 12000 -T 1 -s 0 -c 1000000 -t 0 -g 0 & sleep 0.5s;pkill statsd-tg;sleep 3s;done

statsrelay1.conf

cat statsrelay1.conf
statsd:
  bind: 10.62.4.158:12001
  tcp_cork: false
  validate: true
  shard_map:
    0: 10.62.4.159:12100:udp

ERROR

ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
ERROR: tcpclient[10.62.4.159/12100/udp]: Error from send: Message too long
JeremyGrosser commented 7 years ago

This happens when you try to send metrics to a UDP backend that is larger than your network interface's MTU (probably somewhere around 1500 bytes). You can either use shorter metric keys or switch to TCP for the backend to resolve this problem.

keyboardfann commented 7 years ago

Dear @JeremyGrosser , Because telegraf statsd plugin only supports UDP protocol at this moment, so I need to use UDP protocol. setting max-send-queue: 512MB parameter seems work, please correct me if I am wrong.