kube-HPC / hkube

🐟 High Performance Computing over Kubernetes - Core Repo 🎣
http://hkube.io
MIT License
303 stars 20 forks source link

multiple redundant pipeline drivers are created when there are jobs in queue #1907

Closed golanha closed 2 months ago

golanha commented 3 months ago

HKube micro-service Resource manager determines the number of pipelinedrivers needed according to pipeline driver queue state.

Describe the bug If there are jobs in pipeline driver queue due to cocurrency limit on a certain pipeline, resource manager concludes that there are not enough pipeline drivers and requests reduntant drivers that are stopped after they start.

Expected behavior Resource manager should include the reason the job is in the queue in its calculations.

To Reproduce Steps to reproduce the behavior: create a pipeline with concurrency limit 1 and run it multiple times. see pipelinedriver jobs created and turns to completed right away.