search

wandenberg / nginx-push-stream-module

A pure stream http push technology for your Nginx setup. Comet made easy and really scalable.
Other
2.22k stars 295 forks source link

Random garbage in Event Source #202

Closed subzey closed 9 years ago

subzey commented 9 years ago

Push stream randomly sends random garbage after updating from 0.4.1-rc to 0.5.1 to a client connected by event source.

Here's what I've seen in wireshark:

  1. First comes the packet with some “really” invalid JSON image

Hex dump for your convenience:

0000  00 1d 7d 5c 9e 6b 00 0b  e0 f0 00 ed 08 00 45 00   ..}\.k.. ......E.
0010  00 32 b7 f5 00 00 2f 06  4e a3 32 70 90 07 c0 a8   .2..../. N.2p....
0020  02 0e 00 50 c3 05 ef df  88 3f cf c8 09 76 50 18   ...P.... .?...vP.
0030  00 6e 77 ea 00 00 35 0d  0a 3a 20 2d 31 0a 0d 0a   .nw...5. .: -1...
  1. The second packet arrives that somehow resembles the template, but the payload is 0xA4 image

The hexdump:

0000  00 1d 7d 5c 9e 6b 00 0b  e0 f0 00 ed 08 00 45 00   ..}\.k.. ......E.
0010  00 5e b7 f6 00 00 2f 06  4e 76 32 70 90 07 c0 a8   .^..../. Nv2p....
0020  02 0e 00 50 c3 05 ef df  88 49 cf c8 09 76 50 18   ...P.... .I...vP.
0030  00 6e 40 c9 00 00 33 30  0d 0a 64 61 74 61 3a 20   .n@...30 ..data: 
0040  7b 22 69 64 22 3a 38 38  32 31 31 2c 22 63 68 61   {"id":88 211,"cha
0050  6e 6e 65 6c 22 3a 22 70  75 62 6c 69 63 22 2c 22   nnel":"p ublic","
0060  74 65 78 74 22 3a a4 7d  0a 0a 0d 0a               text":.} ....
  1. And afterwards a perfectly valid message arrives: image

I'm pretty sure we're not sending a 0xA4 chunk, as the message body is generated with LUA's cjson module (ngx.request.capture_multi is used). While debugging I was JSON-decoding any message right before it is sent to push stream and got no errors — looks like LUA part is okay.

Another nuance that may help to isolate the issue. We're using 4 nginx machines in prod, and each public message is sent to each instance. If we send 1000 messages, all 1000 arrives to the client safe and sound, and garbled messages are spread in between with no apparent pattern.

In other words, those garbage messages (most probably) aren't corrupted messages we sent. They just came out of nowhere.

wandenberg commented 9 years ago

@subzey Thanks for the report. This kind of issue are very difficult to reproduce locally. Can we work together to solve it? If yes, contact me on private. In the mean time I will check what are the differences on the two tags that may cause this problem. Can you send me the exact commits that have good and bad result?

wandenberg commented 9 years ago

Fixed on 47ec083e0079d3eb9a96b52d9f58e8f54e2e8e3d

subzey commented 9 years ago

Many thanks!