search

kube-HPC / hkube

🐟 High Performance Computing over Kubernetes - Core Repo 🎣
http://hkube.io
MIT License
305 stars 20 forks source link

stateful block stopped in the middle of streaming pipeline due to "ZMQListener poll timeout" #1837

Closed ism55ism55 closed 8 months ago

ism55ism55 commented 9 months ago

Reported by Yael:

HKube micro-service worker / wrapper

Describe the bug while streaming data to a pipeline job, stateful block gets stopped after several "ZMQListener poll timeout" warnings from wrapper. After stop block still gets msgs from bus (but cannot process them correctly) and pod does not exit.

Expected behavior if gets "ZMQListener poll timeout" which leads to stop also exit

To Reproduce Steps to reproduce the behavior: streaming pipeline stateful blocks run data through pipeline

golanha commented 8 months ago

Happned due to intensive, up and down of stateless. Too many sockets are created. A certain fix in wrapper created the problem, since wrapper 2.5 DEV10 wrapper, problem is fixed