server instability when client connections drop

[WestleyArgentum]

Hey! A group of us were messing around with Troop earlier (it's a really cool tool!) and noticed that when a client's internet connection is interrupted while they are connected to a Troop server then the server enters a strange state where it doesn't crash but also won't function any longer. I've taken a few minutes to test and I can reproduce the error by doing as follows:

1. Start a Troop server on a remote machine (passing log=True, debug=true to the server init. Do this inside a "screen" session or in some way that you will be able to review stdout if your terminal session gets disconnected.

2. Connect a client machine as "user1" (in this case we've been using --mode SonicPi)

3. Run something as a test (play 66) to make sure everything works. Also note server side stdout "New connected user 'user1' from ..."

4. Abruptly cut off wifi to the client, or in some way ungracefully interrupt the clients connection.

5. Attempt to connect a new client "user2".

At that point I consistently get a problem where the editor launches for the new client but it is blank except for their curser and the cursers of all the previous clients that have had their connections interrupted. Nothing can be typed and clicking "evaluate code" has no effect.

In stdout on the server side I see only a new message "New connected user 'user2' from ...", and in Troop/logs/XXX.txt I either see nothing or a few lines from the test like

0.4964 '<3><13><1><9><0>'
0.4973 '<3><14><1><10><0>'
0.5003 '<3><15><1><9><0>'
0.5022 '<2><16><1><[8, -1, 1]><0>'
0.5031 '<2><17><1><[7, -1, 1]><1>'
0.5064 '<2><18><1><[7, "6", 1]><2>'
0.5074 '<2><19><1><[8, "6", 1]><3>'
0.5148 '<7><20><1><1><5><1>'
0.5381 '<3><21><1><10><1>'
0.5394 '<3><22><1><10><1>'

The server is still up, it has not crashed, but at this point the only way to use Troop again is to completely stop and start the server.

Have you seen anything like this before? And / or do you have any advice for the best places to instrument the code with debugging statements?

Thanks a ton for any advice!

[Qirky]

Hey, I've added some keep-alive messages sent between client and servers to better handle when clients are disconnected in weird ways - can you let me know if this improves things?

[Qirky]

Any update?

[Qirky]

@WestleyArgentum have you got any updates on performance? I've added keep alive messages but some other users are reporting more frequent 'kicking out' by the Troop server so I'd like to check if the keep alive system works for you or not. Thanks.