Regression in 0.5.5: Websockets get lost in Server Mode

[icarito]

Hi Manatlan, thanks for your hard work. I upgraded to 0.5.5 in my main instance (https://super-8.educa.juegos/) and started experiencing problems with unreplied websocket requests when in (remote) server mode. At first I thought it was some kind of race condition or network issue but then I found this traceback in the logs:

[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR: Uncaught exception GET /Super8-7f8436495370-ws (127.0.0.1) 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR: HTTPServerRequest(protocol='http', host='super-8.educa.juegos', method='GET', uri='/Super8-7f8436495370-ws', version='HTTP/1.1', remote_ip='127.0.0.1') 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR: Traceback (most recent call last): 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/web.py", line 1703, in _execute 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     result = await result 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/websocket.py", line 278, in get 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     await self.ws_connection.accept_connection(self) 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/websocket.py", line 881, in accept_connection 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     await self._accept_connection(handler) 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/websocket.py", line 964, in _accept_connection 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     await self._receive_frame_loop() 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/websocket.py", line 1118, in _receive_frame_loop 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     await self._receive_frame() 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/websocket.py", line 1209, in _receive_frame 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     await handled_future 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/ioloop.py", line 743, in _run_callback 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     ret = callback() 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/tornado/websocket.py", line 658, in <lambda>
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     self.stream.io_loop.add_future(result, lambda f: f.result()) 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:   File "/home/icarito/.local/lib/python3.8/site-packages/guy.py", line 290, in on_message 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR:     instance = WebSocketHandler.clients[self] 
[12/Apr/2020:22:44:13 +0200] [upstream] [2861224] STDERR: KeyError: <guy.WebSocketHandler object at 0x7f05c0631dc0> 

And on a hunch I downgraded back to 0.5.3 (because I had tested well with it) and the app is back to normal. I'm not sure how I can help you test this. The issue appears when loading games from the cartriges list. More often than not, with 0.5.5, it will fail. With 0.5.3 it seems to work reliably. My code is https://gitlab.com/fuentelibre/super-8

[manatlan]

quick reply (coz I can't test now)

basically, server had received a message (on the socket), from an unknown client. (ex: the client was destroyed just before the reception).

I'd like to know, if this kind of thing could resolve your trouble

replace instance = WebSocketHandler.clients[self] by: instance = WebSocketHandler.clients.get(self,None) if instance is None: return

btw, I can't understand why version>0.5.3 will fail (no big changes on the communication protocol)

[icarito]

Thank you for your reply. I will try your patch and report back. Indeed it's a mystery, I think I've observed the same issue in 0.5.3 but much less. Perhaps my flaky Internet connection that's been a bit unreliable lately. Maybe we should add some sort of disconnection event to guy.js, to let the client deal with a broken websocket?

El dom, 12 de abr de 2020 a las 14:38, manatlan notifications@github.com escribió:

quick reply (coz I can't test now)

basically, server had received a message (on the socket), from an unknown client. (ex: the client was destroyed just before the reception).

I'd like to know, if this kind of thing could resolve your trouble

replace instance = WebSocketHandler.clients[self] by: instance = WebSocketHandler.clients.get(self,None) if instance is None: return

btw, I can't understand why version>0.5.3 will fail (no big changes on the communication protocol)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/manatlan/guy/issues/14#issuecomment-612679614, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABQYSZMUEX7DMFQ7GBZ5ZDRMIYFXANCNFSM4MGRXTEQ.

[icarito]

Hi!

Thanks for your hotfix. I was able to patch v0.5.6 with exactly those two lines and it seems to be working reliably now. I can confirm 0.5.3 worked reliably as well, installing 0.5.5 or 0.5.6 broke it (until your fix).

Didn't try with 0.5.4.

I'll make a PR for you in gratitude. :-)

El dom, 12 de abr de 2020 a las 21:08, Sebastian Silva sebastian@fuentelibre.org escribió:

Thank you for your reply. I will try your patch and report back. Indeed it's a mystery, I think I've observed the same issue in 0.5.3 but much less. Perhaps my flaky Internet connection that's been a bit unreliable lately. Maybe we should add some sort of disconnection event to guy.js, to let the client deal with a broken websocket?

El dom, 12 de abr de 2020 a las 14:38, manatlan notifications@github.com escribió:

quick reply (coz I can't test now)

basically, server had received a message (on the socket), from an unknown client. (ex: the client was destroyed just before the reception).

I'd like to know, if this kind of thing could resolve your trouble

replace instance = WebSocketHandler.clients[self] by: instance = WebSocketHandler.clients.get(self,None) if instance is None: return

btw, I can't understand why version>0.5.3 will fail (no big changes on the communication protocol)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/manatlan/guy/issues/14#issuecomment-612679614, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABQYSZMUEX7DMFQ7GBZ5ZDRMIYFXANCNFSM4MGRXTEQ.