Ran a scenario from 10 min with input rate of 100 msg per sec.
got the following exception during the test run:
wrapper::INFO::WebsocketClient::got message from worker: serviceDiscoveryUpdate
Exception in thread serviceDiscoveryUpdateThread:
Traceback (most recent call last):
File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner
self.run()
File "/usr/lib/python3.9/threading.py", line 892, in run
self._target(self._args, self._kwargs)
File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/wrapper/algorunner.py", line 236, in handle
self._discovery_update(data)
File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/wrapper/algorunner.py", line 346, in _discovery_update
self.streamingManager.setupStreamingListeners(
File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/streaming/StreamingManager.py", line 54, in setupStreamingListeners
listener = MessageListener(options, nodeName)
File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/streaming/MessageListener.py", line 12, in init
self.adapater = ZMQListener(remoteAddress, self.onMessage, self._encoding, receiverNode)
File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/zmq/streaming/ZMQListener.py", line 20, in init
self._worker = self._worker_socket(remoteAddress)
File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/zmq/streaming/ZMQListener.py", line 26, in _worker_socket
worker = context.socket(zmq.DEALER)
File "/opt/venv/lib/python3.9/site-packages/zmq/sugar/context.py", line 235, in socket
s = self._socket_class(self, socket_type, kwargs)
File "/opt/venv/lib/python3.9/site-packages/zmq/sugar/socket.py", line 58, in init
super(Socket, self).init(a, **kw)
File "zmq/backend/cython/socket.pyx", line 328, in zmq.backend.cython.socket.Socket.init
zmq.error.ZMQError: Too many open files
Happens due to intensive change of statelss pods. Happned in the bulk feature use.
Bulk feature has been reverted, it will not happen any more. closing the issue
Ran a scenario from 10 min with input rate of 100 msg per sec. got the following exception during the test run:
wrapper::INFO::WebsocketClient::got message from worker: serviceDiscoveryUpdate Exception in thread serviceDiscoveryUpdateThread: Traceback (most recent call last): File "/usr/lib/python3.9/threading.py", line 954, in _bootstrap_inner self.run() File "/usr/lib/python3.9/threading.py", line 892, in run self._target(self._args, self._kwargs) File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/wrapper/algorunner.py", line 236, in handle self._discovery_update(data) File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/wrapper/algorunner.py", line 346, in _discovery_update self.streamingManager.setupStreamingListeners( File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/streaming/StreamingManager.py", line 54, in setupStreamingListeners listener = MessageListener(options, nodeName) File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/streaming/MessageListener.py", line 12, in init self.adapater = ZMQListener(remoteAddress, self.onMessage, self._encoding, receiverNode) File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/zmq/streaming/ZMQListener.py", line 20, in init self._worker = self._worker_socket(remoteAddress) File "/opt/venv/lib/python3.9/site-packages/hkube_python_wrapper/communication/zmq/streaming/ZMQListener.py", line 26, in _worker_socket worker = context.socket(zmq.DEALER) File "/opt/venv/lib/python3.9/site-packages/zmq/sugar/context.py", line 235, in socket s = self._socket_class(self, socket_type, kwargs) File "/opt/venv/lib/python3.9/site-packages/zmq/sugar/socket.py", line 58, in init super(Socket, self).init(a, **kw) File "zmq/backend/cython/socket.pyx", line 328, in zmq.backend.cython.socket.Socket.init zmq.error.ZMQError: Too many open files