Large Message Stream Reading

[Warfley]

Currently messages are read to the end before being used. They are copied multiple times during that process.

This might be a problem when handling large messages. Maybe a stream accessor would be nice, to read them piecwise, as they arrive and not having to have them stored fully in memory before processing. E.g. for large data uploads which should be written directly to the disk this might become a problem.

[Warfley]

This issue won't be targeted soon, as the client can:

1. Simply send the data in multiple submessages
2. Use a second HTTP channel for uploading and only send control messages via websocket

[esvignolo]

Hi Warfley, thanks for te component!, I have a issue when i connect to android web socket server, we need write a json (346 characters), and the server close de connection before end. When I use the html sample (chatClient.html) connecting to the same server, and sending the same message, is received ok. I try sending as text, raw... even sniffing the communication, I can't know where the problem is. Maybe the size of message?

Thanks!

[Warfley]

I try sending as text, raw... even sniffing the communication, I can't know where the problem is. Maybe the size of message?

I don't think so, I've tested multiple hundered byte messages, and as long as it fits in the RAM this shouldn't be the problem.

The fact that the server simply closes the connection means something unexpected happend. For example the standard defines that if the message does not contain correctly formatted UTF-8 strings, the stream should be closed immediately. Do you use any special (non ansi) characters in your message? If so you need to make sure the string is UTF-8 encoded. The browser works fully on UTF-8, therefore this is not a problem when using the HTML file

For further debugging the websockets protocol supports close messages. This library does currently not support this feature explicetly, but you can set a breakpoint at https://github.com/Warfley/LazWebsockets/blob/master/src/wsstream.pas#L392 You should then be able to use the debugger to read from buffer as string. Another option would be to insert before this line something like this:

        SetLength(str, buffLen);
        if buffLen > 0 then
          Move(buffer, str[1], buffLen);
        writeLn(str);

to print out that message on the console (or do something else with it). This of course might require recompiling the package (If it is used, the example client directly imports the units)

[esvignolo]

Hi! the message was cut, to 127 char. I made a fix and now works, and make a pull request.

[billbarsch]

Hi! the message was cut, to 127 char. I made a fix and now works, and make a pull request.

@esvignolo can you share your fix to us? my connections keeps closing with large messages. Tanks!

[esvignolo]

Hi! the last version in the trunk has the fix already.

[billbarsch]

@esvignolo unfortunately not yet, the same message works in the example client in html and javascipt, but when it is sent with lazarus, the connection keeps closing

do you know any way i could debug this to understand what's going on?

small messages work perfectly, I would also like to thank you for this magnificent work!

[esvignolo]

All credits is for Warfley! Is great solution for use websockets in fpc. The code for solve this problem are posted in Commits on Apr 1, 2020.

Can u check have this changes?

[Warfley]

There is still an underlying problem with fragmented messages. To transmit a long message, simply set the fragment size to be the total size of the message:

with  Communicator.WriteMessage(wmtString, Length(AVeryLargeString)) do
try
  WriteRaw(AVeryLargeString);
finally
  Free;
end;

The first argument of WriteMessage says what type of message you want to send (wmtString, wmtBinary or wmtPing), and the second argument then gives the framentation length. If it is exactly the size of the string (or longer), no fragmentation will happen, which is the source of the bug.

Currently I have no idea what causes the bug, but as fragmentation isn't used that often, it is rather easy to circumvent

[billbarsch]

@Warfley

There is still an underlying problem with fragmented messages. To transmit a long message, simply set the fragment size to be the total size of the message:

with  Communicator.WriteMessage(wmtString, Length(AVeryLargeString)) do
try
  WriteRaw(AVeryLargeString);
finally
  Free;
end;

The first argument of WriteMessage says what type of message you want to send (wmtString, wmtBinary or wmtPing), and the second argument then gives the framentation length. If it is exactly the size of the string (or longer), no fragmentation will happen, which is the source of the bug.

Currently I have no idea what causes the bug, but as fragmentation isn't used that often, it is rather easy to circumvent

this worked perfectly! I just sent a message of over 4 mb!

thank you! very happy here!

hugs from Brazil!

[lainz]

Hi, I have a JSON message about 100 kb of data (sent with a broadcast in a pascal program). The client (another PC with a pascal program) can't connect when the message is big.

But it works fine on a browser with javascript. I mean I can read perfectly the data sent.

Any ideas?

[lainz]

Hi again, sending a binary json gzipped works better, it supports more data. Let's see if it works for our purposes.

Thanks for the library.

[lainz]

Well tested again, and also with a binary message compressed with gzip there is a size limit... the error is in the client, what I get is an exception here:

function TWebsocketCommunincator.GetOpen: boolean;
begin
  Result := FStream.Open;
end;