Framing broken for huge frames

[GoogleCodeExporter]

When using Chrome 16 as a client I got the error of mismatched frame size on 
large frames untilizing several packets. Just looking at this. ;-/

Original issue reported on code.google.com by mo...@tushino.ru on 19 Dec 2011 at 6:12

[GoogleCodeExporter]

How big are your frame sizes? Maybe an endian issue in the framing. Is it an 
error on the server or in Chrome?

After a quick look in the specs the framing didnt change. Anyway knowing the 
size of the frames is crucial when debugging, since that determines how the 
length is encoded.

Original comment by ch...@devristo.com on 19 Dec 2011 at 6:41

[GoogleCodeExporter]

I've tried to respond with desired protocol version as 12 at the server, as 
suggested by RFCs,
  HTTP/1.1 400 Bad Request
  ...
  Sec-WebSocket-Version: 12
but Google seems not conforming to RFC, and shows the error in logs: Unexpected 
response code: 400. Very ugly people are working there.

Original comment by mo...@tushino.ru on 19 Dec 2011 at 6:46

[GoogleCodeExporter]

MISMATCH:actual=48828 header=48389

There can not be errors in Google ;-). They push their updates head over heels.

Original comment by mo...@tushino.ru on 19 Dec 2011 at 6:49

[GoogleCodeExporter]

Think all frames bigger than 2048 in 1 packet fail. That is the fread size 
after each socket_select.

Original comment by ch...@devristo.com on 19 Dec 2011 at 6:57

[GoogleCodeExporter]

Committed a fix. I am using 
http://www.php.net/manual/en/function.stream-get-meta-data.php to find the 
amount of unread_bytes in the php socket buffer because it seems that 
stream_select wont return a socket when there is still data in its buffer after 
a fread. Which is weird imho

Original comment by ch...@devristo.com on 19 Dec 2011 at 7:23

[GoogleCodeExporter]

However, the manual explicitly states that I should use that value in a script. 
But I dont see another way, without making everything non-blocking.

Original comment by ch...@devristo.com on 19 Dec 2011 at 7:33

[GoogleCodeExporter]

I'm not sure this will fix my problem. Strangely, I'm getting now even a first 
frame with payload length equal to 0. Digging...

Original comment by mo...@tushino.ru on 19 Dec 2011 at 7:51

[GoogleCodeExporter]

Okay, then its a seperate issue I am afraid.

Original comment by ch...@devristo.com on 19 Dec 2011 at 8:52

[GoogleCodeExporter]

Hi, I found what is wrong. This is not related to my fix ;-) and Chrome.
The way how the frames with $len = 127 are handled is incorrect. Instead of 
lines 215/216 of framing.php should be:
   list(,$h, $l) = unpack('N2', $raw);
   $frame->payloadLength = ($l + ($h*0x0100000000));
   $raw = substr($raw,8);

Also line 225 should be:
   $fullLength = min($frame->payloadLength - $frame->actualLength, strlen($raw));

Original comment by mo...@tushino.ru on 20 Dec 2011 at 9:41

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Original comment by ch...@devristo.com on 21 Dec 2011 at 10:17

• Changed title: Framing broken for frames of length 127
• Changed state: Fixed

[GoogleCodeExporter]

Will change those lines immediately!

This whole project was just to educate my self a bit on WebSockets. For now I 
only use phpws to sent small messages, other stuff is barely tested.

So lots of thanks for fixing all these bugs!

Original comment by ch...@devristo.com on 21 Dec 2011 at 10:18

[GoogleCodeExporter]

Original comment by ch...@devristo.com on 21 Dec 2011 at 10:24

• Changed title: Framing broken for huge frames

[GoogleCodeExporter]

Hi,
Nevertheless, I was searching for a WebSockets server based on PHP for 
prototyping of one new system, and your project was the first (after several 
other trials) which really started to work just after deployment. The others 
failed.

I did not think that my system will require such big frames, but it actually 
does. The overall traffic load is expected to be very high.

Original comment by mo...@tushino.ru on 21 Dec 2011 at 12:37

[GoogleCodeExporter]

I'm curious how it holds up with the big frames and high traffic. Currently my 
server rarely has more than 20 users connected. With generally small frames 
(forum posts). 

Original comment by ch...@devristo.com on 21 Dec 2011 at 6:12

[GoogleCodeExporter]

I can't say for sure. I'm sending 1Mb messages in a single connection and it 
gets disconnected after 1-st or 2-nd message. But my copy is slightly different 
to yours. Bugs are everywhere. ;-)

Original comment by mo...@tushino.ru on 22 Dec 2011 at 8:40

[GoogleCodeExporter]

The site i use it on used to be pretty hard on phpws as well. Any specific 
reason for the disconnects?

Original comment by ch...@devristo.com on 22 Dec 2011 at 6:28

[GoogleCodeExporter]

If there are no limitations on Hixie message size (implied by standards or by 
implementation in Safari), then this is a bug somewhere. I need to fix it.

Original comment by mo...@tushino.ru on 22 Dec 2011 at 7:04

[GoogleCodeExporter]

I'm lost with this bug. When sending a single big message, it is passed ok. If 
it is being sent after a couple of other messages (even not so big), it looks 
like data is vanishes somewthere on the sockets layer. That is no error on 
sending server, but the browser receives messages of 0 size.

I have an option to merge some of your changes, but my current code does very 
similar writing cycle as yours:

function _write(&$sock, $msg)
{
  $length = strlen($msg);
  while(true)
  {
    $sent = socket_write($sock, $msg, $length);
    if($sent === false)
    {
      $errorcode = socket_last_error();
      $errormsg = socket_strerror($errorcode);
      echo "SENDING ERROR: $errormsg\n";
      return false;
    }
    if($sent < $length)
    {
      $msg = substr($msg, $sent);
      $length -= $sent;
    }
    else
    {
      return true;
    }
  }
  return false;
}

Could you possibly try to send several successive 1Mb messages in your 
environment? In fact I'm sending dataurls with pictures 
(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABQAAAALBCAIAAACIlZblablabla...) 
and you could add this multimedia function to your chat.

Original comment by mo...@tushino.ru on 22 Dec 2011 at 8:48

[GoogleCodeExporter]

It turned out that smaller messages (say 20K-40K) will eventually lead to the 
same problem, but it requires much more such messages to "suffocate" sending. 
1Mb chunk sticks just in a moment, but 30K chunks stick after a minute or so.

Original comment by mo...@tushino.ru on 22 Dec 2011 at 8:59

[GoogleCodeExporter]

Clever idea the data-urls! I will have a look in the weekend. I have seen a 
similar weird issue with the PHPWS client. Data received was sometimes too 
short, while send was good and Chrome worked fine.

Have you tried wireshark to see what is actually sent / received?

Original comment by ch...@devristo.com on 22 Dec 2011 at 9:36

[GoogleCodeExporter]

No. (Is it built as LSP or on other technology?)

Original comment by mo...@tushino.ru on 22 Dec 2011 at 9:55

[GoogleCodeExporter]

I dont know what you mean with LSP. Wireshark uses a driver to capture all 
network traffic. You can then filter that by port or protocol.

Original comment by ch...@devristo.com on 23 Dec 2011 at 7:19

• Changed state: Accepted

[GoogleCodeExporter]

LSP stands for Layered Service Provider, which is a technology to embed 
something into network stack. There is a bunch of products for intercepting 
traffic.

Original comment by mo...@tushino.ru on 23 Dec 2011 at 9:02

[GoogleCodeExporter]

I'm getting "Warning: socket_write(): unable to write to socket [0]: An 
established connection was aborted by the software in your host machine" during 
an attempt to send a 'big' message (second in a row). Just after that I get 
"Warning: socket_recv(): unable to read from socket" for the same socket, and 
the browser gets disconnected (it receives a zero-length message though, 
according to javascript logs).

This only happens in my custom server module, because it is a little bit more 
complex than the demo. It uses two handlers but they do similar ordinary socket 
things as the demo. Most obviuos difference is that while sending data to a 
client of one handler, still mode data is received from a client of second 
handler. Can this produce a kind of blocking issue? I added a big data into the 
demo.php and it has being sent ok repeatedly. But here this is the single 
lengthy operation at a time.

Original comment by mo...@tushino.ru on 23 Dec 2011 at 9:21

[GoogleCodeExporter]

I think you sent the whole msg at once right? This might go wrong. Because the 
server handles all in series. Ideally create message should create multiple 
frames for long data and queue them for sending. Will try to implement this 
tomorrow.

Original comment by ch...@devristo.com on 24 Dec 2011 at 12:50

[GoogleCodeExporter]

Hey, implemented some queuing. Messages are now sent in packets of 1024 when 
they are not blocking.

In theory this should work much better, but since it is barely tested it might 
contain bugs. See the attachment, will deploy it now on my site and commit 
changes when it works good :)

Original comment by ch...@devristo.com on 24 Dec 2011 at 10:47

Attachments:

• phpws.rar

[GoogleCodeExporter]

It seems relatively stable here, I will commit changes since the rar lacks some 
quick fixes.

Original comment by ch...@devristo.com on 24 Dec 2011 at 11:23

[GoogleCodeExporter]

Is it possible for this google repository to setup downloads with source 
archives? GitHub provides options to download zip or tgz of entire trunk.

Original comment by mo...@tushino.ru on 24 Dec 2011 at 12:43

[GoogleCodeExporter]

Not that i see :( Alternative is to keep a local git repository and pull 
everytime to download.

Shame github has that feature and google doesnt :S

Original comment by ch...@devristo.com on 24 Dec 2011 at 12:56

[GoogleCodeExporter]

Why then Google shows this phrase on the project main page - "Currently no 
downloads are available."? It looks like it is possible to setup downloads.

I have tested your latest version. It has the same problem with my project. As 
soon as a big message arrived from one hand (and more keep arriving), sending 
it to another hand is dropped (browser receives 0 length message).

Original comment by mo...@tushino.ru on 24 Dec 2011 at 1:47

[GoogleCodeExporter]

I can set up downloads, but then I need to package and upload it everytime. 
Which I will do when things are more stable ;)

Weird your issue, same errors also occur with the phpws Client? I didnt try to 
sent more than 80 KB but that worked fine.

Original comment by ch...@devristo.com on 24 Dec 2011 at 1:50

[GoogleCodeExporter]

Hmm 1 MB goes wrong with multiple frames with PHPWS client as well.

Original comment by ch...@devristo.com on 24 Dec 2011 at 1:53

[GoogleCodeExporter]

Closing and opening connection in between seems to work fine. So it is probably 
in .socket or .protocol .

Original comment by ch...@devristo.com on 24 Dec 2011 at 1:57

[GoogleCodeExporter]

For some reason the testcase fails (sending 2*10KB in a row), while the log 
says the following.

Read 8192 bytes
Read 8192 bytes
Read 4104 bytes
Written: 4096 bytes
Written: 4096 bytes
Written: 4096 bytes
Written: 4096 bytes
Written: 4096 bytes
Written: 8 bytes

This all adds up to 0 :S So the data is actually being sent, but not correct 
for some reason.

Original comment by ch...@devristo.com on 24 Dec 2011 at 2:22

[GoogleCodeExporter]

Frames are never completely read:

Written: 4096 bytes
Written: 4096 bytes
Written: 2052 bytes
Written: 4096 bytes
Written: 4096 bytes
Written: 2052 bytes
Read: 8192 bytes 
Current length: 0 / 10240 
Read: 8192 bytes 
Current length: 8188 / 10240 
Current length: 0 / 10240 
Read: 4104 bytes 

Original comment by ch...@devristo.com on 24 Dec 2011 at 2:30

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Okay, seems the client is working now. I can sent 2 10MB messages in a row 
Didnt change anything I think on the server side.

However the console output makes it verrrryyy slow. Wil remove some logging 
lines.

Original comment by ch...@devristo.com on 24 Dec 2011 at 2:55

[GoogleCodeExporter]

It seems sockets should work in non-blocking mode in order to eliminate this 
bottle-neck.

Original comment by mo...@tushino.ru on 24 Dec 2011 at 3:13

[GoogleCodeExporter]

BTW, I don't see where is in new sources the code which split a large message 
into short frames, as you wrote. All seems being sended in a single large frame.

Original comment by mo...@tushino.ru on 24 Dec 2011 at 3:17

[GoogleCodeExporter]

That doesnt happen yet, all it does now is spreading big frames over multiple 
packages to avoid blocking on socket write.

Original comment by ch...@devristo.com on 24 Dec 2011 at 3:24

[GoogleCodeExporter]

Creating multiple frames is easy, we just need to chop up the data and set the 
continuation flag.

What I did up till now just makes sure that the packets are sent only when they 
dont block.

Original comment by ch...@devristo.com on 24 Dec 2011 at 3:26

[GoogleCodeExporter]

Well, I'll try to incorporate the PHPMIO 
(http://thethoughtlab.blogspot.com/2007/04/non-blocking-io-with-php-mio.html) 
into PHPWS. Its demo works very fast on concurrent downloads.

Original comment by mo...@tushino.ru on 24 Dec 2011 at 5:51

[GoogleCodeExporter]

Okay might be a good idea. However did you figure out what goes wrong now? Sure 
its in the networking?

Original comment by ch...@devristo.com on 24 Dec 2011 at 6:41

[GoogleCodeExporter]

As far as sockets are still blocking, it seems receiving hand does not even get 
a chance to read first message completely, as the server is overwhelmed by 
incoming messages from other hand. Perhaps, finding an optimal buffer size 
(obviously much larger than 1024, at least 100000) may help a bit.

Original comment by mo...@tushino.ru on 24 Dec 2011 at 7:54

[GoogleCodeExporter]

Yeah 1KB buffer is way to small for big messages. For me its sufficient, you 
might want to try with a lot bigger buffer :)

Original comment by ch...@devristo.com on 25 Dec 2011 at 10:32

[GoogleCodeExporter]

Hmm if is set non-blocking it works much better. Dont know yet if it breaks 
anything though.

Original comment by ch...@devristo.com on 25 Dec 2011 at 11:29

[GoogleCodeExporter]

Attached the code I am currently testing.

Original comment by ch...@devristo.com on 25 Dec 2011 at 11:48

Attachments:

• phpws.rar

[GoogleCodeExporter]

For my case, client is connected and still never gets a message. I could upload 
my test-project just to make sure it does not have an error on its own. PHPMIO 
is on the way, but not yet completely working.

Original comment by mo...@tushino.ru on 25 Dec 2011 at 2:33

[GoogleCodeExporter]

Done. I mean PHPWS does now work using PHPMIO. One of noticable drawbacks of 
non-blocking sockets is that CPU usage is 100%. I added a conditional sleep in 
the main loop, but this slightly decrease responsiveness.

Nevertheless, my project can not run properly even in this configuration. I 
even changed the order in which sockets are checked: writing in checked before 
reading. Nothing helps. After the first message (only 10K), the process stalls. 
Perhaps, I have a dull error in my project, but I don't see it.

Original comment by mo...@tushino.ru on 25 Dec 2011 at 8:21