Upgrading to 10.3.3 causes "frame cannot uniquely be identified by it's labels" errors for alerts that were previously working fine

moxious / triage

testing triage actions for issues

0 stars 1 forks source link

Upgrading to 10.3.3 causes "frame cannot uniquely be identified by it's labels" errors for alerts that were previously working fine #309

Open tonypowa opened 5 months ago

tonypowa commented 5 months ago

What happened?

We recently attempted to upgrade from 10.2.3 to 10.3.3 for the recent security fix. Unfortunately we quickly saw that this caused errors with multiple alerts and from multiple datasource types. The error is:


Invalid format of evaluation results for the alert definition : frame cannot uniquely be identified by its labels: has duplicate results with labels {}

Revert back to 10.2.3 fixes the issue.

I found another person having the same issue:

https://www.reddit.com/r/grafana/comments/1ay5egy/getting_this_alert_error_in_grafana_1033_what_am/

What did you expect to happen?

alerts to continue to function

Did this work before?

yes

How do we reproduce it?

upgrade to 10.3.3
check alerts for above error

Is the bug inside a dashboard panel?

Environment (with versions)?

Grafana: 10.3.3

OS: mac

Browser: chrome

Grafana platform?

Kubernetes

Datasource(s)?

No response

tonypowa commented 5 months ago

This issue is a test copy of an issue in another repo. Original issue: https://github.com/grafana/grafana/issues/#83448

moxious commented 5 months ago

Hi @tonypowa, it looks like the issue might be related to an alerting bug introduced in the version 10.3.3. As it impacts alert functionality and not a specific datasource, the most suitable project for this issue seems to be the Alerting project. It would be best to raise this problem there so that it can be addressed by the team specializing in alerting features.

moxious commented 5 months ago

Summary: After upgrading to Grafana 10.3.3, multiple alerts from different data sources began failing with errors about frames not being uniquely identifiable by their labels, and reverting to version 10.2.3 resolves the issue.

moxious commented 5 months ago

Elaboration:

Hi @tonypowa, thank you for reporting this issue! In order to fully understand and address the problem you're encountering with Grafana after upgrading to 10.3.3, we need a bit more information. Here are some questions that can aid us in diagnosing the issue more effectively:

Can you list the specific datasource types that you observed this error with? Knowing which datasources are affected can help narrow down the cause.
Could you provide the configuration for one or two of the alerts that are failing? This information might help us spot any potential issues with alert settings that could lead to this error.
Have you applied any custom modifications or plugins to Grafana that might not be directly compatible with the new version?
Are you able to share the snippet of the query that feeds data into the alerting mechanism? You can obfuscate any sensitive information.
Is there any chance you could provide us with a minimal reproducible example or the steps you took before the error showed up, beyond just upgrading Grafana?
Lastly, you mentioned checking the alerts for the above error—could you include detailed steps or screenshots of how you're checking these alerts in the new version?

Once we have this additional information, we'll have a much better chance of pinpointing the issue and providing a solution. Thank you for your cooperation! 🛠️