Open Barteus opened 5 months ago
Thanks for reporting this, @Barteus !
Yes I noticed this as well and we have been discussing this with Observability team in Vancouver. The issue here is that the metrics in prometheus-pushgateway are not removed at the end of the spark job, and as a consequence, prometheus keeps on scraping them.
We are discussing what is the best way to get them removed from pushgateway, whether to have the spark job process to do that at the end (but we would risk to remove them before prometheus would scrape them) or have a process to do so.
I'll keep this thread posted as soon as we come up with a decision for the way forward.
Reproduce
Actual
The Driver pod was removed in the Completed state.
In the dashboards the metric shows that they still use resources:
In Kubernetes there is no Pod, so resources are not used.
Expected
The used resources after completion of the job are 0.
Versions
Operating system: Ubuntu 22.04.3 LTS
Juju CLI: 3.3.1
Juju agent: 3.3.1
Charm revision:
microk8s:
COS:
cos-configuration-k8s config: