search

grafana / k8s-monitoring-helm

Apache License 2.0
128 stars 52 forks source link

Duplicated alert rules for agent and alloy #517

Open rodrigoccaldas23 opened 1 month ago

rodrigoccaldas23 commented 1 month ago

Hello 👋 We recently changed from grafana agent to alloy (v1.0.6) and we have noticed that both alert rules from Grafana Agent and Alloy are now appearing, resulting in duplication. This redundancy could potentially lead to missed alerts or unnecessary notifications. Is there any way to remove the duplicate alert rules associated with Grafana Agent and ensure that only the alert rules specific to Grafana Alloy are active?

image

skl commented 1 month ago

@rodrigoccaldas23 can you please raise a support ticket in Grafana Cloud? The alert and recording rules are not controlled by this helm chart. You can mention this ticket and my name for their reference in case they want to ping me internally 👍

rodrigoccaldas23 commented 1 month ago

Thanks @skl. We've also observed that alongside the alloy service, a Grafana agent service is being created. Is this supposed to occur?

image

skl commented 1 month ago

We've also observed that alongside the alloy service, a Grafana agent service is being created. Is this supposed to occur?

@petewall what do you think to the above? I'm assuming it should be Alloy only.

petewall commented 1 month ago

The reason for the Grafana Agent service is to reduce breaking changes when upgrading from pre 1.0 versions of this Helm chart to 1.0 versions. If you had configured applications to send OpenTelemetry data to Grafana Agent, after the upgrade to 1.0, you would have to reconfigure all of your applications. This service points to the Alloy instances, not to any actual Grafana Agent instances.