Sending protobuf data cause frontend javascript websocket failed: Invalid frame header

[gogolxdong]

readFile() of a protobuf data file sizes 152.6kb , sendText and sendBinary cause frontend javascript websocket failed: Invalid frame header.

[dom96]

I guess this is in general an issue with sending large amounts of data over the websocket?

[gogolxdong]

Reduce to one item data and it remains. I think it's out of the protobuf binary data breaking the websocket frame format.

[gogolxdong]

For websocket frame format reference, https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers

[metagn]

This seems to work:

import websocket, uri, asyncdispatch

const site = "wss://echo.websocket.org:443/".parseUri
const data = char(0x08) & char(0x96) & char(0x01)

let ws = waitFor newAsyncWebsocketClient(site)
waitFor ws.sendBinary(data)
echo waitFor(ws.readData())

It might be a problem with the specific data, maybe you could try sending to echo.websocket.org with the faulty binary and seeing what happens.

[gogolxdong]

@dom96 Your guess is probably right, what is the limit of sending websocket data ?

[gogolxdong]

I tested the boundary is around 35 data items, sizes 12.2 KB.

[gogolxdong]

Do not test with echo.websocket.org which is also limited to 131k strings.

[metagn]

Perhaps you could use makeFrame to see if the produced frame is valid?

[gogolxdong]

proc makeFrame*(opcode: Opcode, data: string, masked: bool): string doesn't make a difference . I noticed it uses fin:true, since the given data is truncated by which , frame data size limit or the browser for some reasons ,and by how much of the data size are both unknown yet, frame is unable to be made manually with a valid data size.

[metagn]

I guess an option for paginating over 10kb could be added if it were to be actually useful. Maybe you could see how something like this works for your case (could be added in a prettier way if it does)

proc paginateSendBinary(ws: AsyncWebSocket, data: string) =
  const pageLen = 10240
  if data == "": return
  let lenm1 = data.len - 1
  let pageAmount = lenm1 div pageLen
  for i in 0..<pageAmount:
    ws.sock.send(makeFrame(Frame(fin: false, rsv1: false, rsv2: false,
      rsv3: false, masked: false, opcode: Opcode.Binary,
      data: data[(i*pageLen) ..< ((i+1)*pageLen))]))
  ws.sock.send(makeFrame(Frame(fin: true, rsv1: false, rsv2: false,
    rsv3: false, masked: false, opcode: Opcode.Binary,
    data: data[(pageAmount*pageLen) .. lenm1])))

[dom96]

There does indeed seem to be some bug when sending large amounts of data at once. Annoying thing is that I'm just seeing a socket closure when this happens, here is the JSON data I was trying to send. I'll try to reproduce it: https://gist.github.com/dom96/9dba05c10f2d22c1c3d49d5d78e5444e

[dom96]

Cannot reproduce it with this:

import websocket, uri, asyncdispatch

const site = "ws://demos.kaazing.com:80/echo".parseUri
const data = readFile("payload.json")

let ws = waitFor newAsyncWebsocketClient(site)
waitFor ws.sendText(data)
let d = waitFor(ws.readData())
echo(d.data.len, " == ", data.len)
echo(d.data == data)

(echo.websocket.org fails... btw why doesn't the websocket client use httpclient?)