K-Phoen
commented
1 year ago Hi,
Consider the following dashboards organization layout (which isn't hypothetical, that's pretty much what we use where I work):
Grafana
├── Service A [Folder]
│ ├── Service A SLI/SLO [Dashboard; tags: service-a, slo]
│ │ ├── Error budget burn rate [Alert; tags: service-a, team: team-a, severity: X]
│ │ └── Remaining error budget [Alert; tags: service-a, team: team-a, severity: X]
│ └── Service A Overview [Dashboard; tags: service-a, overview]
│ ├── Some alert [Alert; tags: service-a, team: team-a, severity: X]
│ └── Some other alert [Alert; tags: service-a, team: team-a, severity: X]
├── Service B [Folder]
│ ├── Service B SLI/SLO [Dashboard; tags: service-b, slo]
│ │ ├── Error budget burn rate [Alert; tags: service-b, team: team-a, severity: X]
│ │ └── Remaining error budget [Alert; tags: service-b, team: team-a, severity: X]
│ └── Service B Overview [Dashboard; tags: service-b, overview]
│ ├── Some alert [Alert; tags: service-b, team: team-a, severity: X]
│ └── Some other alert [Alert; tags: service-b, team: team-a, severity: X]
└── Service C [Folder]
├── Service C SLI/SLO [Dashboard; tags: service-c, slo]
│ ├── Error budget burn rate [Alert; tags: service-c, team: team-b, severity: X]
│ └── Remaining error budget [Alert; tags: service-c, team: team-b, severity: X]
└── Service C Overview [Dashboard; tags: service-c, overview]
├── Some alert [Alert; tags: service-c, team: team-a, severity: X]
└── Some other alert [Alert; tags: service-c, team: team-a, severity: X]The easiest way for me to get all dashboards for a service is by tag. The easiest way for me to get all alerts for a service is by tag.
I understand that dashboards are probably organized and tagged differently in your environment, but that doesn't make querying alerts or dashboards by tags "almost useless" :)
Regards.
What is the reason behind using tags? It's almost useless, consider you have a bunch of boards for K8S monitoring and they all have one tag and you don't have permission to change this but you only need one exact dashboard about pod resources for your services. So when you push this tag to backstage, you get a useless list with many links. And for the alerts - do you have any examples where one use the same tag for dashboard and a couple of alerts? (usually you create more then one alert in Grafana for an object). I guess the best way is to get dashboards by UID, which one can pick up easily and it won't be changed or deleted. And for alerts it's better to have dedicated parameter in config and query them by name. Also both parameters should accept more then one value. Regards.