avito-tech / bioyino

High performance and high-precision multithreaded StatsD server
The Unlicense
228 stars 22 forks source link

Corruption of last char in tag value #63

Closed Karsonito closed 3 years ago

Karsonito commented 3 years ago

The last symbol of the last tag is chaotically replaced

Sent tag values

test_corrupted_tags;y=y;x=x
test_corrupted_tags;y=yyyyyy;x=xxxx
test_corrupted_tags;y=yyy;x=xxx

Received

test_corrupted_tags;x=x;y=x
test_corrupted_tags;x=xxxx;y=yyyyyx
test_corrupted_tags;x=xxx;y=yyx

IMPORTANT: tags must be sent in reversed order (for example y before x)

All versions affected: 0.6, 0.7.2, 0.8.0

It's a little hard to reproduce and depends from some factors Here is config which makes it possible:

bufsize = 1500
multimessage = true
mm-packets = 10
buffer-flush-time = 0
buffer-flush-length = 16384

(It's possible with other values too, but needs more time to happens)

Here some commands to save time and speed up testing

while true; do for x in x xx xxx xxxx xxxxx xxxxxx xxxxxxx xxxxxxxx xxxxxxxxxx; do for y in y yy yyy yyyy yyyyy yyyyyy yyyyyyy yyyyyyyy yyyyyyyyy yyyyyyyyyy; do echo "test_corrupted_tags;y=$y;x=$x:1|c" | nc -u 127.0.0.1 8125; echo -n "."; done; done; echo; done

nc -k -l -p 2003 | grep test_corrupted_tags | grep -v -P "test_corrupted_tags;x=x+;y=y+ "

Please fix )

Albibek commented 3 years ago

I think I've fixed this. At least your case is not reproduced now.

The fix is in the bioyino-metrics library, so bioyino has no changes. To get the fix you need to run cargo update and then rebuild the latest version (0.8.0).

Karsonito commented 3 years ago

I tested new build and all is fine!

Oh yeah! Grand thanks for fix! It takes some days to make it reproducible ) Very complicated factors and relatively rare event even in production.

One more. Not important Redundant semicolon in metric doubles last char of tag value Sent:

test.tags;x=a;:1|c
test.tags;;x=a:1|c

Received:

test.tags;x=a; 1.0 1621684595
test.tags;x=aa 1.0 1621684607

Have a nice weekends!

Albibek commented 3 years ago

Fixed that too. Thanks for finding it. The update is the same as previous one

Karsonito commented 3 years ago

It's a pity, but problem is still here

Compiling bioyino-metric v0.5.0 (https://github.com/Albibek/bioyino-metric?branch=0.5.0#463470bf)
...
test.tags;;x=a:1|c
test.tags;;;x=a:1|c
...
test.tags;x=aa 1.0 1621710848
test.tags;x=a=a 1.0 1621710850
Karsonito commented 3 years ago

Hi! Any news?

Albibek commented 3 years ago

Didn't have lots of time over the week. In the process now. Expecting this to be fixed after weekends.

Albibek commented 3 years ago

Fixed one more place, please check if it's OK now. Update is same as before

Karsonito commented 3 years ago

Great thanks! Finally a win )