Memory overflow from twisted Web-socket server when writing data faster than client can consume.

[s-borah]

I had to deploy a Websocket server to distribute realtime financial market data to clients using autobahn. Initially it worked fine but eventually it started causing RAM overflow in the server and after some painful debugging, found the cause to be a temporary data buffer (transport._tempDataBuffer) from the transport layer in twisted.

I handled the issue by accessing the protected member and clearing the buffer after a certain size while also disconnecting the client (generally caused due to slow internet at client's end).

This is a serious issue though and a proper fix should be implemented at either autobahn or twisted or both. Basically the library should protect the server from accumulating data incase the client is not able to receive it. Guess nobody uses a python web socket server for high speed data streaming anyways so the issue might not have bothered anyone.

To recreate the issue, just send large data very fast while the client has connected but is not receiving the data.

[oberstet]

Guess nobody uses a python web socket server for high speed data streaming anyways so the issue might not have bothered anyone.

No, you didn't RTFM, that's the reason;)

You should use https://twistedmatrix.com/documents/current/core/howto/producers.html

https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/websocket/streaming/streaming_producer_client.py https://github.com/crossbario/autobahn-python/blob/master/examples/twisted/websocket/streaming/streaming_server.py

[s-borah]

I do remember trying to using use the Producer, Consumer flow and the same issue persisted... Will confirm the same and revert back with more information if that's the case.