Large Websocket Responses Triggering Multiple Data Events

[DavidVine]

Update the websocket.axi file to support large responses that come in over multiple data events.

[jackbilesisdm]

HI David,

I've been recently looking into this.

I can confirm that the DATA_EVENT has an inherent memory limit in which it fires multiple events to meet the character array length of the received string. The limit appears to not be of a fixed length as it was reporting different lengths of arrays on the same message sent multiple times to the master.

I had initially looked in the fragmentation of packets in the WebSocket protocol but this did not appear to be the case as the FIN bit was creating a 0 in unexpected places of long strings, so the call of webSocketFrameFromString was creating false data.

The below is my work around:

websocket.axi

define_type
struct WebSocketFrame {
    char fin;
    char rsv1;
    char rsv2;
    char rsv3;
    char opCode;
    char mask;
    char maskingKey[4];
    char payloadData[10000]; // payload length can be determined from calling length_array or lenth_string for payloadData
    integer payloadLength;
}

define_function webSocketFrameFromString(WebSocketFrame wsf, char data[]) {
    ...

    if((data[2] BAND $7F) == 0) {
        payloadLen = 0;
        nByte = 3;
    } else if((data[2] BAND $7F) <= 125) {
        payloadLen = (data[2] BAND $7F);
        nByte = 3;
    } else if((data[2] BAND $7F) == 126) {
        payloadLen = ((data[3] << 8) BOR (data[4]))
        nByte = 5;
    } else if((data[2] BAND $7F) == 127) {
        payloadLen = ((data[3] << 56) BOR (data[4] << 48) BOR (data[5] << 40) BOR (data[6] << 32) 
                                    BOR (data[7] << 24) BOR (data[8] << 16) BOR (data[9] << 8) BOR (data[10]));
        nByte = 11;
    }

    wsf.payloadLength = type_cast(payloadLen)

    print("'PAYLOAD LENGTH SHOUDL BE -> ',payloadLen",false)
    if(wsf.mask) {
        wsf.maskingKey = "data[nByte],data[nByte+1],data[nByte+2],data[nByte+3]";
        wsf.payloadData = webSocketUnmask(mid_string(data,nByte+4,payloadLen),wsf.maskingKey);
    } else {
        wsf.payloadData = mid_string(data,nByte,payloadLen);
    }
}

DEFINE_VARIABLE
startBuffer = False
WebSocketFrame thisFrame

define_event data_event[wsSockets] {

    online: {
        ...
    }

    offline: {
        ...
    }

    onerror: {
               ...
    }

    string: {
        integer idx;
        dev socket;
        stack_var HttpResponse response; //stack_var HttpMessage httpResponse;
        stack_var WebSocketFrame wsf;

        idx = get_last(wsSockets)
        socket = wsSockets[idx];

        print("'Received data (',itoa(length_array(data.text)),' bytes) on Socket[',devToString(socket),'].'",false);
        if((webSockets[idx].readyState == OPEN) || (webSockets[idx].readyState == CLOSING)) { // assume we received a websocket protocol string

            if(startBuffer == False){
                startBuffer = True
                webSocketFrameFromString(wsf,data.text);
                print('PAYLOAD START', false)
                thisFrame = wsf
            }
            else if(startBuffer == True){
                print('BUILDING PAYLOAD',false)
                thisFrame.payloadData = "thisFrame.payloadData , data.text"
            }

            if(length_array(thisFrame.payloadData) >= thisFrame.payloadLength){
                print('PAYLOAD COMPLETE', false)
                print( "'PAYLOAD LEGNTH -> ', thisFrame.payloadLength",false)
                if(thisFrame.mask) {
                    webSocketOnMessage(socket,webSocketUnmask(wsf.payloadData,wsf.maskingKey));
                } else {
                    webSocketOnMessage(socket,thisFrame.payloadData);
                }

                startBuffer = False
            }
                  ...

           }
    }
}

The above appears to work for a singular WebSocket connection but i have not intended on testing with multiple sockets running. Granted it's a quick job but it gets the point across about what i'm trying to achieve.

[DavidVine]

Hi @jackbilesisdm,

Apologies it took me a year and a half to get around to actually fixing this. There was a lot happening in my life at the start of last year and then this just fell off my radar completely. I was looking through GitHub today for the first time in ages and thought "I should probably fix that".

I've re-written the buffer processing in websockets.axi to allow for processing of very large incoming data which might be processed over multiple data events.

I'm not sure what the upper-limit is but I've successfully tested processing of incoming websocket data with a payload of up to 1 million characters which comes in over 819 data events and gets stitched back together by websockets.axi and triggers the webSocketOnMessage handler 62 times. The programmer would then have to write their own buffer processing to work with the multiple calls to webSocketOnMessage.