[BUG] RuntimeError: cannot schedule new futures after interpreter shutdown

[cos-gl]

Describe the bug Just installed latest release of the TB Gateway usign apt packet. Configured from scratch. Created OPCUA connector with no issues in the communication between TB server and Gateway but cannot TB-GW is not contacting the OPCUA server. It's not a network problem (TB-GW, TB server and OPC server are on the same lan, ping is ok, ports are reachable, no firewall). Tried with type OPCUA and also OPCUA ASYNCIO, same result. The output from "systemctl status thingsboard-gateway" are costantly saying "RuntimeError: cannot schedule new futures after interpreter shutdown" (I'll list you below the complete log) but I didn't see anything in the log files...

If I start the tb-gateway with the command

/usr/bin/python3 -c "from thingsboard_gateway.tb_gateway import daemon; daemon()" 

I see the same erros below, shown in the std output.

I attack you the relevant config files tb-gateway-config.zip

Connector name (If bug in the some connector): OPC-UA and OPC-UA ASYNCIO Connector

Error traceback (If available):

2024-06-21 12:55:51 - |ERROR| - [tb_logger.py] - tb_logger - exception - 127 - cannot schedule new futures after interpreter shutdown
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/thingsboard_gateway/connectors/opcua/opcua_connector.py", line 175, in start_client
    async with self.__client:
  File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/client.py", line 94, in __aenter__
    await self.connect()
  File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/client.py", line 305, in connect
    await self.connect_socket()
  File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/client.py", line 369, in connect_socket
    await self.uaclient.connect_socket(self.server_url.hostname, self.server_url.port)
  File "/var/lib/thingsboard_gateway/.local/lib/python3.12/site-packages/asyncua/client/ua_client.py", line 300, in connect_socket
    await asyncio.wait_for(asyncio.get_running_loop().create_connection(self._make_protocol, host, port), self._timeout)
  File "/usr/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 1080, in create_connection
    infos = await self._ensure_resolved(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 1456, in _ensure_resolved
    return await loop.getaddrinfo(host, port, family=family, type=type,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 901, in getaddrinfo
    return await self.run_in_executor(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/asyncio/base_events.py", line 863, in run_in_executor
    executor.submit(func, *args), loop=self)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 172, in submit
    raise RuntimeError('cannot schedule new futures after '
RuntimeError: cannot schedule new futures after interpreter shutdown

Versions:

• OS: Ubuntu 24.04 LTS
• Thingsboard IoT Gateway version: 3.5.1-1
• Thingsboard server: 3.6.4-1
• Python version: 3.12

[imbeacon]

Hi @cos-gl,

Thank you for information, you are right, this issue appears if you restart the connector, could you try to use a master branch version and let us know about the results? We have added a fix. Probably, we will create a hotfix release on next week.

[cos-gl]

Hi @imbeacon, sorry for the delay in this reply. I will take a look in the next days and I will let you know. In the meantime I was using gateway on docker (as a test) and I didn't experienced this behaviour. Thanks

[cos-gl]

Hi, just updated from master branch. The gateway doesn't reconnect to the OPC server and after a complete restart (OPC server, thingsboard server, thingsboard gateway) the gateway logs shown this

{"log":"2024-06-27 06:36:36 - |INFO| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _process_config_request - 204 - Configuration update request received.\r\n","stream":"stdout","time":"2024-06-27T06:36:36.432180021Z"}
{"log":"2024-06-27 06:39:19 - |WARNING| - [tb_device_mqtt.py] - tb_device_mqtt - _on_disconnect - 343 - MQTT client was disconnected with reason code 7 (The connection was lost.) \r\n","stream":"stdout","time":"2024-06-27T06:39:19.589941921Z"}
{"log":"2024-06-27 06:39:49 - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 352 - MQTT client \u003cpaho.mqtt.client.Client object at 0x72fc227f44d0\u003e - Connected!\r\n","stream":"stdout","time":"2024-06-27T06:39:49.997990973Z"}
{"log":"2024-06-27 06:39:49 - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 236 - Subscribed to *|* with id 2 for device *\r\n","stream":"stdout","time":"2024-06-27T06:39:49.998595612Z"}
{"log":"2024-06-27 06:42:05 - |INFO| - [tb_gateway_remote_configurator.py] - tb_gateway_remote_configurator - _process_config_request - 204 - Configuration update request received.\r\n","stream":"stdout","time":"2024-06-27T06:42:05.75746973Z"}
{"log":"2024-06-27 06:42:05 - |ERROR| - [tb_gateway_service.py] - tb_gateway_service - __connect_with_connectors - 872 - ['33089995-5dbc-40a9-ac72-c9a4165ac8f6'] Error on loading connector 'test_connector_GL': RuntimeError(\"There is no current event loop in thread 'Remote Request Processing'.\")\r\n","stream":"stdout","time":"2024-06-27T06:42:05.762057989Z"}
{"log":"2024-06-27 06:42:05 - |ERROR| - [tb_logger.py] - tb_logger - exception - 127 - There is no current event loop in thread 'Remote Request Processing'.\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767821097Z"}
{"log":"Traceback (most recent call last):\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767847074Z"}
{"log":"  File \"/thingsboard_gateway/gateway/tb_gateway_service.py\", line 857, in __connect_with_connectors\r\n","stream":"stdout","time":"2024-06-27T06:42:05.76785169Z"}
{"log":"    connector = self._implemented_connectors[connector_type](self,\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767855207Z"}
{"log":"                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767870459Z"}
{"log":"  File \"/thingsboard_gateway/connectors/opcua/opcua_connector.py\", line 81, in __init__\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767873497Z"}
{"log":"    current_loop = asyncio.get_event_loop()\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767875648Z"}
{"log":"                   ^^^^^^^^^^^^^^^^^^^^^^^^\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767935449Z"}
{"log":"  File \"/usr/local/lib/python3.11/asyncio/events.py\", line 681, in get_event_loop\r\n","stream":"stdout","time":"2024-06-27T06:42:05.767939309Z"}
{"log":"    raise RuntimeError('There is no current event loop in thread %r.'\r\n","stream":"stdout","time":"2024-06-27T06:42:05.768140747Z"}
{"log":"RuntimeError: There is no current event loop in thread 'Remote Request Processing'.\r\n","stream":"stdout","time":"2024-06-27T06:42:05.768157543Z"}

I really don't understand the meaning... The telemetry wasn't collected since yesterday (when I restarted TB server and TB gateway).

EDIT: after a TB gateway service restart the connection with the OPC server started and telemetry is now collected (but I'm sure I restarted the gateway also before, I completely rebooted the machine)

[imbeacon]

We have improved OPC-UA connector and in the latest version it works more stable, please try to use it.