websockets / ws

Simple to use, blazing fast and thoroughly tested WebSocket client and server for Node.js
MIT License
21.61k stars 2.42k forks source link

Is there anyway to disable the websocket closeTimeout? #2203

Closed Mohan-Vetri closed 6 months ago

Mohan-Vetri commented 7 months ago

Is there an existing issue for this?

Description

I conducted a load test on the WS package. Within 5 minutes, this load test triggered over 4000 socket connections in total, closing each socket after one second. It concurrently triggered a maximum of 15 socket connections per second. I noticed that some WebSockets took 30 seconds to emit the close event on the server, even though the client had closed the socket before that. The WebSocket's ReadyState was 2 (indicating it's in the closing state). Why is there a 30-second timer to emit the close event when the server knows the socket is closed on the client end and has the close code & reason?

Reason why I am looking for this option: In production, we anticipate receiving a high volume of WebSocket connections, approximately 300+ per second. During peak hours, we expect to have up to 70,000 active socket connections. We also have a ping pong interval of 60 seconds in place to terminate lingering connections. Even if we reduce the interval to 30 seconds, it will still occupy server resources for that duration. If the server takes 30 seconds to emit the close event, it could result in lingering 'zombie' connections, unnecessarily occupying server resources. Additionally, since we save WebSocket objects in memory to facilitate message broadcasting, and only remove the WebSocket object from memory on the close event, this delay could lead to further performance overheads on the server.

Is there any option to disable the closeTimeout? Is there any other solution?

Thank you!

ws version

8.16.0

Node.js Version

v10.24.1 and v18.18.2

System

No response

Expected result

No response

Actual result

No response

Attachments

No response

lpinca commented 7 months ago

Why is there a 30-second timer to emit the close event when the server knows the socket is closed on the client end and has the close code & reason?

Because a misbehaving client might keep the connection open indefinitely. When the connection is closed (correctly or forcibly) the timeout is cleared.

Mohan-Vetri commented 7 months ago

Thanks for the response. The connection is closed properly on the client end. On the Server side, We are checking the readyState of the WebSocket before sending a message to it. The readyState of the WebSocket is 2. The server knows that it was closed and it has closeCode (_closeCode) as well. As I mentioned before, those connections will be there on the server for 30 seconds, even though they are in closing state, and the server will allocate resources for those lingering connections for that duration. Could you please suggest a solution to overcome this issue?

lpinca commented 7 months ago

The connection is closed properly on the client end. On the Server side, We are checking the readyState of the WebSocket before sending a message to it. The readyState of the WebSocket is 2. The server knows that it was closed and it has closeCode (_closeCode) as well.

It means the connection is not closed. The socket did not emit the 'close' event, otherwise the timeout is cleared. See

https://github.com/websockets/ws/blob/d343a0cf7bba29a4e14217cb010446bec8fdf444/lib/websocket.js#L1283

Mohan-Vetri commented 7 months ago

Thanks for the clarification. May I know why there is a delay in emitting the close event when we clear the closeTimer while setting the readyState of the WebSocket to 2 on SocketOnClose function? Sometimes it took 3 seconds, 10 seconds, or even 30 seconds to emit the close event of the socket, which is in the closing state.

lpinca commented 7 months ago

The 'close' event on the websocket is emitted after the 'close' event is emitted on the underlying socket and after all buffered data is read.

Mohan-Vetri commented 7 months ago
lpinca commented 7 months ago

Please correct me if I am mistaken, I believe that receiverOnConclude is the method that will be invoked first, providing the closeCode and reason when the client closes the WebSocket connection

That is only one part of the closing handshake. It means a peer received the close frame. The connection can be closed only when a peer has both sent and received a close frame.

Could we please consider performing the cleanup tasks there and terminating the underlying socket (which is in closing state), thereby emitting the close event right away?

You can use websocket.terminate() to do that but the connection is abruptly closed without completing the closing handshake.

Do we need to wait until the buffered data is read, and would terminating the socket in the receiverOnConclude method cause any issues?

All buffered data is read regardlessly, but you have no guaranteed that the close handshake completes correctly if you destroy the socket.

We frequently encounter situations where multiple sockets linger in the closing state for over 10 seconds, sometimes up to 30 seconds as mentioned earlier.

You should monitor the 'close' event of the underlying socket. Check if that is emitted correctly.

It would be beneficial if the server could promptly identify when a socket enters this closing state, close the underlying connection, and emit the close event accordingly

It does, that's why the close timer exists. If the connection is not closed correctly in a reasonable amount of time, it is forcibly closed.

These lingering 'zombie' connections unnecessarily consume server resources, which could potentially impact the performance of the server.

That is probably a network / overload system issue.

lpinca commented 7 months ago

See

https://github.com/websockets/ws/blob/d343a0cf7bba29a4e14217cb010446bec8fdf444/lib/websocket.js#L269-L282

lpinca commented 6 months ago

I'm closing this as answered.