calaldees / KaraKara

Karaoke Event System - Attendees can view and queue tracks from their mobile phones
https://karakara.uk
GNU General Public License v3.0
28 stars 9 forks source link

website: mosquito sockets failing to close #90

Closed shish closed 2 years ago

shish commented 2 years ago

After working fine for a few days, this appeared in the website logs, and all calls to the website got a generic "internal error" response from nginx. Restarting the website container made it work again.

website_1        | Traceback (most recent call last):
website_1        |   File "/usr/local/bin/pserve", line 8, in <module>
website_1        |     sys.exit(main())
website_1        |   File "/usr/local/lib/python3.10/site-packages/pyramid/scripts/pserve.py", line 30, in main
website_1        |     return command.run()
website_1        |   File "/usr/local/lib/python3.10/site-packages/pyramid/scripts/pserve.py", line 281, in run
website_1        |     server(app)
website_1        |   File "/usr/local/lib/python3.10/site-packages/paste/deploy/loadwsgi.py", line 193, in server_wrapper
website_1        |     return fix_call(context.object,
website_1        |   File "/usr/local/lib/python3.10/site-packages/paste/deploy/util.py", line 55, in fix_call
website_1        |     val = callable(*args, **kw)
website_1        |   File "/usr/local/lib/python3.10/site-packages/waitress/__init__.py", line 23, in serve_paste
website_1        |     serve(app, **kw)
website_1        |   File "/usr/local/lib/python3.10/site-packages/waitress/__init__.py", line 19, in serve
website_1        |     server.run()
website_1        |   File "/usr/local/lib/python3.10/site-packages/waitress/server.py", line 322, in run
website_1        |     self.asyncore.loop(
website_1        |   File "/usr/local/lib/python3.10/site-packages/waitress/wasyncore.py", line 245, in loop
website_1        |     poll_fun(timeout, map)
website_1        |   File "/usr/local/lib/python3.10/site-packages/waitress/wasyncore.py", line 172, in poll
website_1        |     r, w, e = select.select(r, w, e, timeout)
website_1        | ValueError: filedescriptor out of range in select()
shish commented 2 years ago

It looks like every time the website sends an MQTT command, it opens a new connection and never closes it, so eventually we run out of sockets

shish commented 2 years ago

Should be fixed by 6d552bc - since paho.Client() automatically handles request queuing / threading / reconnecting, we could actually make one global Client object rather than opening and closing connections every time

shish commented 2 years ago

api_queue opens one MQTT connection in @app.listener('before_server_start'), so this shouldn't happen