Description
We have a long lived application which periodically writes and reads tags, and sometimes communicates on "demand" (when changing params on GUI app). Everything works without issues, but after some time (40mins - couple of days) we suddenly see a lot of errors and "on demand" communication no longer works.
On "demand" which was mentioned, works such that we connect to PLC, write tags, and close connection (all done using context manager). Periodic writes and reads are done in separate threads, having their own connections which are constantly utilized - no issues there.
When we encounter errors, restarting application solves problem, everything works for some time and then eventually issues come back.
# first example of error
[ ControllerProcess-Thread-2 (_syste] 09-30 08:41:51 ERROR: plc.py::from_plc()::212: Failed to get PLC info
Traceback (most recent call last):
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 328, in get_plc_info
raise ResponseError(f"get_plc_info did not return valid data - {response.error}")
pycomm3.exceptions.ResponseError: get_plc_info did not return valid data - Failed to parse reply - Error packing -50 as USINT
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\******\backend\src\comm\plc.py", line 205, in from_plc
with self._plc as comm:
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
self.open()
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
self._initialize_driver(**self._init_args)
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 174, in _initialize_driver
self._info = self.get_plc_info()
^^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 336, in get_plc_info
raise ResponseError("Failed to get PLC info") from err
pycomm3.exceptions.ResponseError: Failed to get PLC info
[ ControllerProcess-MainThread ] 09-30 08:41:51 ERROR: plc.py::to_plc()::176: failed to get attribute list
Traceback (most recent call last):
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 499, in _get_instance_attribute_list_service
raise ResponseError(
pycomm3.exceptions.ResponseError: send_unit_data returned not valid data - Failed to parse reply - Error packing -116 as USINT
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\******\backend\src\comm\plc.py", line 168, in to_plc
with self._plc as comm:
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
self.open()
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
self._initialize_driver(**self._init_args)
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 192, in _initialize_driver
self.get_tag_list(program="*" if init_program_tags else None)
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 81, in wrapped
return func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 422, in get_tag_list
tags = self._get_tag_list()
^^^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 438, in _get_tag_list
all_tags = self._get_instance_attribute_list_service(program)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 511, in _get_instance_attribute_list_service
raise ResponseError("failed to get attribute list") from err
pycomm3.exceptions.ResponseError: failed to get attribute list
[ ControllerProcess-MainThread ] 09-30 08:41:51 ERROR: plc.py::to_plc()::176: Failed to get PLC info
Traceback (most recent call last):
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 328, in get_plc_info
raise ResponseError(f"get_plc_info did not return valid data - {response.error}")
pycomm3.exceptions.ResponseError: get_plc_info did not return valid data - Failed to parse reply - Error packing -128 as USINT
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\******\backend\src\comm\plc.py", line 168, in to_plc
with self._plc as comm:
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
self.open()
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
self._initialize_driver(**self._init_args)
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 174, in _initialize_driver
self._info = self.get_plc_info()
^^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 336, in get_plc_info
raise ResponseError("Failed to get PLC info") from err
pycomm3.exceptions.ResponseError: Failed to get PLC info
Those errors keeps going forever, when finally we receive another type of error
# second example of error
[ ControllerProcess-Thread-2 (_syste] 09-30 09:00:00 ERROR: plc.py::from_plc()::212: failed to send message
Traceback (most recent call last):
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\socket_.py", line 53, in send
sent = self.sock.send(msg[total_sent:])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 587, in _send
self._sock.send(message)
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\socket_.py", line 58, in send
raise CommError("socket connection broken.") from err
pycomm3.exceptions.CommError: socket connection broken.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "D:\******\backend\src\comm\plc.py", line 205, in from_plc
with self._plc as comm:
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 144, in __enter__
self.open()
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 165, in open
self._initialize_driver(**self._init_args)
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 171, in _initialize_driver
target_identity = self._list_identity()
^^^^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 264, in _list_identity
response = self.send(request)
^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\logix_driver.py", line 1387, in send
return super().send(request)
^^^^^^^^^^^^^^^^^^^^^
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 574, in send
self._send(request.build_request(**request_kwargs))
File "D:\******\backend\.venv\Lib\site-packages\pycomm3\cip_driver.py", line 589, in _send
raise CommError("failed to send message") from err
pycomm3.exceptions.CommError: failed to send message
Unfortunately we tried different techniques (ex reseeding message) but eventually "on demand" communication is blocked entirely and only restart helps
Pre-checks
Description We have a long lived application which periodically writes and reads tags, and sometimes communicates on "demand" (when changing params on GUI app). Everything works without issues, but after some time (40mins - couple of days) we suddenly see a lot of errors and "on demand" communication no longer works.
On "demand" which was mentioned, works such that we connect to PLC, write tags, and close connection (all done using context manager). Periodic writes and reads are done in separate threads, having their own connections which are constantly utilized - no issues there.
When we encounter errors, restarting application solves problem, everything works for some time and then eventually issues come back.
Target PLC Model: 1769-L18ER-BB1B Firmware Revision: 32.015
Logs
Those errors keeps going forever, when finally we receive another type of error
Unfortunately we tried different techniques (ex reseeding message) but eventually "on demand" communication is blocked entirely and only restart helps