Cos proxy is sending to prometheus targets, from only one prometheus-target relation

canonical / cos-proxy-operator

https://charmhub.io/cos-proxy

Apache License 2.0

2 stars 12 forks source link

Cos proxy is sending to prometheus targets, from only one prometheus-target relation #143

Open err404r opened 5 months ago

err404r commented 5 months ago

Bug Description

We have several telegraf apps related to prometheus-target interface of cos-proxy. On prometheus we could see data only from first telegraf unit. As well as within relation data On cos proxy unit I could see data from relations to all telegraf units

To Reproduce

Relate several telegraf units to one cos-proxy

Environment

Charm revision 74

Relevant log output

Unfortunately no logs to add

Additional context

No response

sed-i commented 5 months ago

Does the deployment look something like this?

graph LR

subgraph LXD
telegraf ---|http| cos-proxy
telegraf ---|prometheus-rules| cos-proxy
end

subgraph COS
prometheus
end

cos-proxy ---|prometheus_scrape| prometheus

err404r commented 4 months ago

yes, but we have several telegraf applications related to one cos-proxy

lucabello commented 2 months ago

We should try to reproduce this issue with a few telegraf deployments. It might be just a timing issue, with cos-proxy taking a long time to settle with too many telegraf deployments.