search

sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.12k stars 1.29k forks source link

monitoring: add metric for open file descriptors and alert when approaching fd limit #10009

Open beyang opened 4 years ago

beyang commented 4 years ago

Exceeding the limit on open file descriptors causes major issues. Some environments have extremely low limits by default (we document how to raise the fd limit here). We should track this metric and alert when it approaches the limit.

bobheadxi commented 4 years ago

% fd used: https://sourcegraph.com/-/debug/grafana/explore?orgId=1&left=%5B%22now-1h%22,%22now%22,%22Prometheus%22,%7B%22expr%22:%22process_open_fds%7Bapp%3D%5C%22gitserver%5C%22%7D%20%2F%20process_max_fds%7Bapp%3D%5C%22gitserver%5C%22%7D%20*%20100%22%7D,%7B%22ui%22:%5Btrue,true,true,%22none%22%5D%7D%5D

can we use this to replace fs inodes alert? https://sourcegraph.com/-/debug/grafana/d/prometheus/prometheus?viewPanel=12&orgId=1&from=now-7d&to=now seems a lot more useful than the strange high threshold we have on that one now

github-actions[bot] commented 3 years ago

Heads up @davejrt @ggilmore @dan-mckean @caugustus-sourcegraph @stephanx - the "team/delivery" label was applied to this issue.