sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.1k stars 1.28k forks source link

Sourcegraph.com: noisy alert: warning_*_frontend_internal_api_error_responses #12450

Closed slimsag closed 4 years ago

slimsag commented 4 years ago

This alert:

"warning_gitserver_frontend_internal_api_error_responses"

Is too noisy because there are regular "real" downtime events of the frontends on each deploy / commit to master in this repository:

image

https://sourcegraph.com/-/debug/grafana/d/gitserver/git-server?orgId=1&from=1594950546998&to=1595403156172

Will need to figure out how to adjust this and then unsilence the alert: https://github.com/sourcegraph/deploy-sourcegraph-dot-com/blob/e33d7cd48e9407aac88124eec89644dd4d51699c/base/frontend/sourcegraph-frontend.ConfigMap.yaml#L5271-L5275

bobheadxi commented 4 years ago

This might be related to https://github.com/sourcegraph/sourcegraph/issues/12158

bobheadxi commented 4 years ago

The new ratio version of this alert no longer fires: https://sourcegraph.com/-/debug/grafana/d/gitserver/git-server?viewPanel=9&orgId=1&from=now-7d&to=now , it indicates a very low percentage of requests erroring out