kube-HPC / hkube

🐟 High Performance Computing over Kubernetes - Core Repo 🎣
http://hkube.io
MIT License
305 stars 20 forks source link

Mark tasks in warning status in etcd in graph #1850

Closed golanha closed 8 months ago

golanha commented 8 months ago

Incase of a coplete failure of redis, worker exists turning tasks status to warning. But since all compenents contiuability relay on redis, if all replicas of redis fail and some times even if just some replicas do fail, there is not recovery and the graph apears as if its still running. The solusion is to create new cleaner in gc-service that would update the graph node by the tasks status

golanha commented 8 months ago

duplcate #1854