search

linux-automation / lxa-iobus

Linux Automation IOBus server
https://lxa-iobus.readthedocs.io/en/latest/
Apache License 2.0
3 stars 4 forks source link

Hanging IOBus Server at 100% CPU load #47

Open Emantor opened 4 weeks ago

Emantor commented 4 weeks ago

Today I observed a hanging IOBus Server with 100% one core load:

Jun 12 14:38:49 rlabD-srv systemd[1]: Starting lxa-iobus.service - LXA iobus Server...
Jun 12 14:38:49 rlabD-srv systemd[1]: Started lxa-iobus.service - LXA iobus Server.
Jun 12 14:38:53 rlabD-srv lxa-iobus-server[1225]: starting server on http://*:8080/
Jun 14 08:38:47 rlabD-srv lxa-iobus-server[1225]: WARNING:lxa-iobus.network:lss_ping: node <LxaBusNode(address=00000507.00000002.00000003.00000002, node_id=1)> does not respond
Jun 14 08:39:19 rlabD-srv lxa-iobus-server[1225]: WARNING:lxa-iobus.network:lss_ping: node <LxaBusNode(address=00000507.00000002.00000003.00000005, node_id=2)> does not respond
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ERROR:aiohttp-json-rpc.server:list.remove(x): x not in list
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: Traceback (most recent call last):
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 325, in _handle_rpc_msg
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     await self._ws_send_str(http_request, result)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 278, in _ws_send_str
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     self.clients.remove(client)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ValueError: list.remove(x): x not in list
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ERROR:asyncio:Task exception was never retrieved
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: future: <Task finished name='Task-1938381' coro=<JsonRpc._handle_rpc_msg() done, defined at /usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py:283> exception=Val>
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: Traceback (most recent call last):
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 325, in _handle_rpc_msg
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     await self._ws_send_str(http_request, result)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 278, in _ws_send_str
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     self.clients.remove(client)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ValueError: list.remove(x): x not in list
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: During handling of the above exception, another exception occurred:
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: Traceback (most recent call last):
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 339, in _handle_rpc_msg
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     await self._ws_send_str(http_request, encode_error(
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 278, in _ws_send_str
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     self.clients.remove(client)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ValueError: list.remove(x): x not in list
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ERROR:aiohttp-json-rpc.server:list.remove(x): x not in list
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: Traceback (most recent call last):
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 325, in _handle_rpc_msg
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     await self._ws_send_str(http_request, result)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 278, in _ws_send_str
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     self.clients.remove(client)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ValueError: list.remove(x): x not in list
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ERROR:asyncio:Task exception was never retrieved
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: future: <Task finished name='Task-1938382' coro=<JsonRpc._handle_rpc_msg() done, defined at /usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py:283> exception=Val>
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: Traceback (most recent call last):
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 325, in _handle_rpc_msg
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     await self._ws_send_str(http_request, result)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 278, in _ws_send_str
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     self.clients.remove(client)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ValueError: list.remove(x): x not in list
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: During handling of the above exception, another exception occurred:
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: Traceback (most recent call last):
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 339, in _handle_rpc_msg
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     await self._ws_send_str(http_request, encode_error(
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:   File "/usr/ptx-venvs/lxa-iobus/lib/python3.11/site-packages/aiohttp_json_rpc/rpc.py", line 278, in _ws_send_str
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]:     self.clients.remove(client)
Jun 17 06:24:30 rlabD-srv lxa-iobus-server[1225]: ValueError: list.remove(x): x not in list

This looks like some issue related to client handling?

SmithChart commented 2 weeks ago

aiohttp-json-rpc is discontinued. So we should think about removing it completely anyway.

I would like to keep this issue open. Maybe we can collect more occurrences - or at least let other know - until we find the time to remove aiohttp-json-rpc from this project.