ccfos / nightingale

An all-in-one observability solution which aims to combine the advantages of Prometheus and Grafana. It manages alert rules and visualizes metrics, logs, traces in a beautiful web UI.
https://flashcat.cloud/docs/
Apache License 2.0
9.86k stars 1.43k forks source link

loki alert silence some labels #2172

Closed lddlww closed 2 months ago

lddlww commented 2 months ago

when used loki for alert,people my want to known what message result the alert, i used loki metric query like

sum by (host_ip,level,message,type) (count_over_time(({level=~"error|crit|emerg|warning|err"} |~ "NIC"|~"down" |json message="message")[5m]))>=1

but this query will generate many alerts in one time,because the label of message is difference ; instead it using

sum by (host_ip,level,type) (count_over_time(({level=~"error|crit|emerg|warning|err"} |~ "NIC"|~"down" |json message="message")[5m]))>=1

, which generate one alert in one time,this could be better, but people could not found more detail,as message missing

how can i do to solve above problem?

best wishes

UlricQin commented 2 months ago

The second approach is better. Once the engineer receives the alert, they can then query for detailed information. Displaying log details within the alert event is not a best practice.