If the config gets updated and one target gets removed, the last prometheus metric for the deleted target remains as the latest prometheus metrics.
New targets:
https://google.com
Prometheus metrics:
sparrow_latency_duration_seconds{status="200",target="https://google.com"} 0.014013145 <- This remains and is not cleaned up
sparrow_latency_duration_seconds{status="200",target="https://example.com"} 0.007915321
Expected Behavior
Lets discuss how (if) we should handle this case.
Maybe it's possible to remove metrics from the prometheus registry
Steps To Reproduce
Deploy sparrow with two targets for any check using http loader
Let it run for a check cycle
Remove one target from the config
HTTP GET /metrics
Relevant logs and/or screenshots, environment information, etc.
Screenshot shows what happened after google.com was removed from the config, compared to a target that is currently active
Since the sparrow metrics api still exposes the value for google.com, prometheus stores it on every scrape
Who can address the issue?
@puffitos @y-eight @lvlcn-t Let's discuss how we handle this next week
Is there an existing issue for this?
Current Behavior
Sparrow starts with two targets:
this creates two prometheus for those targets:
If the config gets updated and one target gets removed, the last prometheus metric for the deleted target remains as the latest prometheus metrics. New targets:
Prometheus metrics:
Expected Behavior
Lets discuss how (if) we should handle this case.
Maybe it's possible to remove metrics from the prometheus registry
Steps To Reproduce
Relevant logs and/or screenshots, environment information, etc.
Screenshot shows what happened after google.com was removed from the config, compared to a target that is currently active
Since the sparrow metrics api still exposes the value for google.com, prometheus stores it on every scrape
Who can address the issue?
@puffitos @y-eight @lvlcn-t Let's discuss how we handle this next week
Anything else?
No response