search

kube-HPC / hkube

🐟 High Performance Computing over Kubernetes - Core Repo 🎣
http://hkube.io
MIT License
305 stars 20 forks source link

when starting a pipeline some blocks are stuck on `creating` until stopping and starting again #1852

Open ism55ism55 opened 8 months ago

ism55ism55 commented 8 months ago

Reported by Yael:

HKube micro-service version 2.6.20, task executer

Describe the bug when starting a pipeline some blocks are stuck on creating status for a long time (above 10 minutes) until stopping the pipeline and starting again.

Expected behavior all blocks should be successfully created in a reasonable time

To Reproduce Steps to reproduce the behavior: streaming pipeline stateful and stateless blocks (including min stateless pods definition) start a pipeline, monitor via jobs dashboard