kube-HPC / hkube

🐟 High Performance Computing over Kubernetes - Core Repo 🎣
http://hkube.io
MIT License
307 stars 20 forks source link

Unneeded pods of algorithms with hotworkers stay up. #1773

Open golanha opened 1 year ago

golanha commented 1 year ago

If an algorithm has a positive value in it's minhotworkers attribute, in case there were more requests for algorithm the actually needed, it won't exit and stay up forever.

To reproduce, create an kubernetes algorithm job manully (while no hkube job will consume it) you;ll see that in case no minhotworkers is set , a pod will be created and then exit, incase there is minhotworkers the created pod will stay running.

half solution would be, to write to log incase worker is up and has no work. maybe check in kubernetes how many jobs of that algorithm already exist and compare with the number of hot workers configured.