search

parse-community / parse-server

Parse Server for Node.js / Express
https://parseplatform.org
Apache License 2.0
20.9k stars 4.78k forks source link

[LiveQuery]RangeError: Invalid WebSocket frame: invalid payload length 126 #6413

Open JeffGuKang opened 4 years ago

JeffGuKang commented 4 years ago

Related to #6173

Issue Description

RangeError: Invalid WebSocket frame: invalid payload length 126 This error occurs intermittently and stops the server when I use livequery.

Steps to reproduce

  1. Run livequery by Parse Server
  2. Use livequery
  3. It happens irregularly

Expected Results

Parse Server should not terminate by the error.

Actual Outcome

Parse Server is terminated. 

RangeError: Invalid WebSocket frame: invalid payload length 126
    at Receiver.getInfo (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:213:16)
    at Receiver.startLoop (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:127:22)
    at Receiver._write (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:74:10)
    at doWrite (_stream_writable.js:435:12)
    at writeOrBuffer (_stream_writable.js:419:5)
    at Receiver.Writable.write (_stream_writable.js:309:11)
    at Socket.socketOnData (/var/app/current/node_modules/parse-server/node_modules/ws/lib/websocket.js:875:35)
    at Socket.emit (events.js:223:5)
    at Socket.EventEmitter.emit (domain.js:475:20)
    at addChunk (_stream_readable.js:309:12)
/var/app/current/node_modules/parse-server/lib/ParseServer.js:229
          throw err;
          ^

RangeError: Invalid WebSocket frame: invalid payload length 126
    at Receiver.getInfo (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:213:16)
    at Receiver.startLoop (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:127:22)
    at Receiver._write (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:74:10)
    at doWrite (_stream_writable.js:435:12)
    at writeOrBuffer (_stream_writable.js:419:5)
    at Receiver.Writable.write (_stream_writable.js:309:11)
    at Socket.socketOnData (/var/app/current/node_modules/parse-server/node_modules/ws/lib/websocket.js:875:35)
    at Socket.emit (events.js:223:5)
    at Socket.EventEmitter.emit (domain.js:475:20)
    at addChunk (_stream_readable.js:309:12)
Emitted 'error' event on WebSocket instance at:
    at Receiver.receiverOnError (/var/app/current/node_modules/parse-server/node_modules/ws/lib/websocket.js:780:13)
    at Receiver.emit (events.js:223:5)
    at Receiver.EventEmitter.emit (domain.js:475:20)
    at errorOrDestroy (internal/streams/destroy.js:108:12)
    at onwriteError (_stream_writable.js:450:5)
    at onwrite (_stream_writable.js:477:5)
    at Receiver.startLoop (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:148:5)
    at Receiver._write (/var/app/current/node_modules/parse-server/node_modules/ws/lib/receiver.js:74:10)
    at doWrite (_stream_writable.js:435:12)
    at writeOrBuffer (_stream_writable.js:419:5) {
  [Symbol(status-code)]: 1002
}

Environment Setup

dplewis commented 4 years ago

Is there a way you can view the frames or debug the socket? What client are you using?

JeffGuKang commented 4 years ago

Is there a way you can view the frames or debug the socket? What client are you using?

I use default socket on parse server. 

const api = new ParseServer({
  databaseURI,
  cloud: __dirname + '/src/cloud/', // location after tsc
  appId: process.env.APP_ID,
  masterKey: process.env.MASTER_KEY,

  serverURL: 'http://localhost:1337/parse',
  // enableAnonymousUsers: true,
  liveQuery: {
    classNames: ['A', 'B', 'C', 'D'],
  },
  auth: {    apple: {
      client_id: process.env.APPLE_SIGNIN_CLIENT_ID, // optional (for extra validation), use the Service ID from Apple.
    },
  },
  maxUploadSize: '5mb',
  logLevel: process.env.NODE_ENV !== 'local' ? 'warn' : undefined,
})

And I did not debug frames of the socket. I will share it after digging.

dplewis commented 4 years ago

At least we can prevent the server from crashing when an error occurs, even if we can't replicate it.

JeffGuKang commented 4 years ago

@dplewis I got a error message through the latest PR(#6418)

error: Invalid WebSocket frame: invalid payload length 126
error: {
  "_events": {
    "close": [
      null,
      null
    ]
  },
  "_eventsCount": 3,
  "readyState": 2,
  "protocol": "",
  "_binaryType": "nodebuffer",
  "_closeFrameReceived": false,
  "_closeFrameSent": false,
  "_closeMessage": "",
  "_closeTimer": null,
  "_closeCode": 1002,
  "_extensions": {},
  "_receiver": {
    "_writableState": {
      "objectMode": false,
      "highWaterMark": 16384,
      "finalCalled": false,
      "needDrain": false,
      "ending": false,
      "ended": false,
      "finished": false,
      "destroyed": false,
      "decodeStrings": true,
      "defaultEncoding": "utf8",
      "length": 0,
      "writing": false,
      "corked": 0,
      "sync": true,
      "bufferProcessing": false,
      "writecb": null,
      "writelen": 0,
      "afterWriteTickInfo": null,
      "bufferedRequest": null,
      "lastBufferedRequest": null,
      "pendingcb": 0,
      "prefinished": false,
      "errorEmitted": true,
      "emitClose": true,
      "autoDestroy": false,
      "bufferedRequestCount": 0,
      "corkedRequestsFree": {
        "next": null,
        "entry": null
      }
    },
    "writable": true,
    "_events": {},
    "_eventsCount": 6,
    "_binaryType": "nodebuffer",
    "_extensions": {},
    "_maxPayload": 104857600,
    "_bufferedBytes": 650,
    "_buffers": [
      {
        "type": "Buffer",
        "data": [
          2, 
...(VERY LONG),
          125
        ]
      }
    ],
    "_compressed": false,
    "_payloadLength": 126,
    "_mask": {
      "type": "Buffer",
      "data": [
        84,
        126,
        124,
        222
      ]
    },
    "_fragmented": 0,
    "_masked": true,
    "_fin": true,
    "_opcode": 10,
    "_totalPayloadLength": 0,
    "_messageLength": 0,
    "_fragments": [],
    "_state": 0,
    "_loop": false
  },
  "_sender": {
    "_extensions": {},
    "_socket": {
      "connecting": false,
      "_hadError": false,
      "_parent": null,
      "_host": null,
      "_readableState": {
        "objectMode": false,
        "highWaterMark": 16384,
        "buffer": {
          "head": null,
          "tail": null,
          "length": 0
        },
        "length": 0,
        "pipes": null,
        "pipesCount": 0,
        "flowing": true,
        "ended": false,
        "endEmitted": false,
        "reading": false,
        "sync": false,
        "needReadable": true,
        "emittedReadable": false,
        "readableListening": false,
        "resumeScheduled": false,
        "paused": false,
        "emitClose": false,
        "autoDestroy": false,
        "destroyed": false,
        "defaultEncoding": "utf8",
        "awaitDrain": 0,
        "readingMore": false,
        "decoder": null,
        "encoding": null
      },
      "readable": true,
      "_events": {
        "end": [
          null,
          null
        ]
      },
      "_eventsCount": 4,
      "_writableState": {
        "objectMode": false,
        "highWaterMark": 16384,
        "finalCalled": false,
        "needDrain": false,
        "ending": false,
        "ended": false,
        "finished": false,
        "destroyed": false,
        "decodeStrings": false,
        "defaultEncoding": "utf8",
        "length": 0,
        "writing": false,
        "corked": 0,
        "sync": false,
        "bufferProcessing": false,
        "writecb": null,
        "writelen": 0,
        "afterWriteTickInfo": null,
        "bufferedRequest": null,
        "lastBufferedRequest": null,
        "pendingcb": 0,
        "prefinished": false,
        "errorEmitted": false,
        "emitClose": false,
        "autoDestroy": false,
        "bufferedRequestCount": 0,
        "corkedRequestsFree": {
          "next": {
            "next": null,
            "entry": null
          },
          "entry": null
        }
      },
      "writable": true,
      "allowHalfOpen": true,
      "_sockname": null,
      "_pendingData": null,
      "_pendingEncoding": "",
      "server": {
        "_events": {},
        "_eventsCount": 5,
        "_connections": 532,
        "_handle": {},
        "_usingWorkers": false,
        "_workers": [],
        "_unref": false,
        "allowHalfOpen": true,
        "pauseOnConnect": false,
        "httpAllowHalfOpen": false,
        "timeout": 120000,
        "keepAliveTimeout": 5000,
        "maxHeadersCount": null,
        "headersTimeout": 40000,
        "_connectionKey": "4:null:1337"
      },
      "_server": {
        "_events": {},
        "_eventsCount": 5,
        "_connections": 532,
        "_handle": {},
        "_usingWorkers": false,
        "_workers": [],
        "_unref": false,
        "allowHalfOpen": true,
        "pauseOnConnect": false,
        "httpAllowHalfOpen": false,
        "timeout": 120000,
        "keepAliveTimeout": 5000,
        "maxHeadersCount": null,
        "headersTimeout": 40000,
        "_connectionKey": "4:null:1337"
      },
      "timeout": 0,
      "parser": null,
      "_paused": false,
      "_httpMessage": null,
      "_peername": {
        "address": "::ffff:127.0.0.1",
        "family": "IPv6",
        "port": 54414
      }
    },
    "_firstFragment": true,
    "_compress": false,
    "_bufferedBytes": 0,
    "_deflating": false,
    "_queue": []
  },
  "_socket": {
    "connecting": false,
    "_hadError": false,
    "_parent": null,
    "_host": null,
    "_readableState": {
      "objectMode": false,
      "highWaterMark": 16384,
      "buffer": {
        "head": null,
        "tail": null,
        "length": 0
      },
      "length": 0,
      "pipes": null,
      "pipesCount": 0,
      "flowing": true,
      "ended": false,
      "endEmitted": false,
      "reading": false,
      "sync": false,
      "needReadable": true,
      "emittedReadable": false,
      "readableListening": false,
      "resumeScheduled": false,
      "paused": false,
      "emitClose": false,
      "autoDestroy": false,
      "destroyed": false,
      "defaultEncoding": "utf8",
      "awaitDrain": 0,
      "readingMore": false,
      "decoder": null,
      "encoding": null
    },
    "readable": true,
    "_events": {
      "end": [
        null,
        null
      ]
    },
    "_eventsCount": 4,
    "_writableState": {
      "objectMode": false,
      "highWaterMark": 16384,
      "finalCalled": false,
      "needDrain": false,
      "ending": false,
      "ended": false,
      "finished": false,
      "destroyed": false,
      "decodeStrings": false,
      "defaultEncoding": "utf8",
      "length": 0,
      "writing": false,
      "corked": 0,
      "sync": false,
      "bufferProcessing": false,
      "writecb": null,
      "writelen": 0,
      "afterWriteTickInfo": null,
      "bufferedRequest": null,
      "lastBufferedRequest": null,
      "pendingcb": 0,
      "prefinished": false,
      "errorEmitted": false,
      "emitClose": false,
      "autoDestroy": false,
      "bufferedRequestCount": 0,
      "corkedRequestsFree": {
        "next": {
          "next": null,
          "entry": null
        },
        "entry": null
      }
    },
    "writable": true,
    "allowHalfOpen": true,
    "_sockname": null,
    "_pendingData": null,
    "_pendingEncoding": "",
    "server": {
      "_events": {},
      "_eventsCount": 5,
      "_connections": 532,
      "_handle": {},
      "_usingWorkers": false,
      "_workers": [],
      "_unref": false,
      "allowHalfOpen": true,
      "pauseOnConnect": false,
      "httpAllowHalfOpen": false,
      "timeout": 120000,
      "keepAliveTimeout": 5000,
      "maxHeadersCount": null,
      "headersTimeout": 40000,
      "_connectionKey": "4:null:1337"
    },
    "_server": {
      "_events": {},
      "_eventsCount": 5,
      "_connections": 532,
      "_handle": {},
      "_usingWorkers": false,
      "_workers": [],
      "_unref": false,
      "allowHalfOpen": true,
      "pauseOnConnect": false,
      "httpAllowHalfOpen": false,
      "timeout": 120000,
      "keepAliveTimeout": 5000,
      "maxHeadersCount": null,
      "headersTimeout": 40000,
      "_connectionKey": "4:null:1337"
    },
    "timeout": 0,
    "parser": null,
    "_paused": false,
    "_httpMessage": null,
    "_peername": {
      "address": "::ffff:127.0.0.1",
      "family": "IPv6",
      "port": 54414
    }
  },
  "_isServer": true
}
dplewis commented 4 years ago

Thanks for the feedback. Did your server crash?

We can try to use the following to reverse engineer the payload that is failing. Knowing _buffers.data would be nice, I don't know if would help.

Also we should show the following for VERBOSE=1.

"_receiver": {
    "_writableState": {
      ...
    },
    "writable": true,
    "_events": {},
    "_eventsCount": 6,
    "_binaryType": "nodebuffer",
    "_extensions": {},
    "_maxPayload": 104857600,
    "_bufferedBytes": 650,
    "_buffers": [
      {
        "type": "Buffer",
        "data": [
          2, 
...(VERY LONG),
          125
        ]
      }
    ],
    "_compressed": false,
    "_payloadLength": 126,
    "_mask": {
      "type": "Buffer",
      "data": [
        84,
        126,
        124,
        222
      ]
    },
    "_fragmented": 0,
    "_masked": true,
    "_fin": true,
    "_opcode": 10,
    "_totalPayloadLength": 0,
    "_messageLength": 0,
    "_fragments": [],
    "_state": 0,
    "_loop": false
  },
JeffGuKang commented 4 years ago

@dplewis My EC2 server did not crash anymore in v4.0.x. 😄 But sometimes the EC2's CPU load reached 100%. I am not sure it is related to this issue or not. (parse server v4.0.1). I will rollback and check my server with v3.10.0 again today.

I will attach the more samples to files.

nodejserror.json.zip

majidhassan commented 3 years ago

Hello, I'm getting a similar issue with the livequery server (parse server & parse livequery server versions are 2.8.4 - yes, I know quite outdated XD )

I'm connecting to the livequery server from my Angular components and I sometimes get this error below, which then crashes the server and sends it into a restart loop since pm2 attempts to restart, which could last anywhere between 5 minutes to an hour after which it resolves by itself.

Any ideas what this error means and how to solve it, or at least how to stop it from crashing the server without having to upgrade to a more recent parse-server version for the livequery server?

If not, any idea if it's possible to run different versions of parse for the main server and the livequery server?

RangeError: Invalid WebSocket frame: RSV1 must be clear
    at Receiver.getInfo (/home/parse/node_modules/parse-server/node_modules/ws/lib/receiver.js:194:9)
    at Receiver.startLoop (/home/parse/node_modules/parse-server/node_modules/ws/lib/receiver.js:152:16)
    at Receiver.add (/home/parse/node_modules/parse-server/node_modules/ws/lib/receiver.js:138:10)
    at Socket._ultron.on (/home/parse/node_modules/parse-server/node_modules/ws/lib/websocket.js:143:54)
    at Socket.emit (events.js:182:13)
    at Socket.EventEmitter.emit (domain.js:441:20)
    at addChunk (_stream_readable.js:287:12)
    at readableAddChunk (_stream_readable.js:268:11)
    at Socket.Readable.push (_stream_readable.js:223:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:122:17)
/home/parse/node_modules/parse-server/lib/ParseServer.js:218
          throw err;
mtrezza commented 3 years ago

@majidhassan Please note that Parse does not currently offer Long Terms Support and we only address issues that are reproducible with latest version of Parse Server.

If not, any idea if it's possible to run different versions of parse for the main server and the livequery server?

LiveQuery is a feature of Parse Server, not an individual repository. Therefore I would generally recommend to use a single version of that repository. Mixing instances of different Parse Server versions using the same DB may or may not work. It is not officially documented as supported, therefore the compatibility is undefined and not recommended for production environments.

majidhassan commented 3 years ago

@mtrezza thanks for your reply, and I understand that it's an older version. However, any ideas where to start on that error?

mtrezza commented 3 years ago

To confirm that this is an issue we'd need to be able to reproduce this with the latest version of Parse Server and without mixing different versions of Parse Server. Then we can handle the issue here.

If you need support for your custom deployment with older versions of Parse Server and/or mixed up Parse Server versions, you would post a topic in the Parse community forum or on StackOverflow. Code-level support is not provided in GitHub issues.