search

sourcegraph / sourcegraph-public-snapshot

Code AI platform with Code Search & Cody
https://sourcegraph.com
Other
10.1k stars 1.28k forks source link

It is not possible to alert on gitserver memory usage, possibly affects other Go services #12685

Open slimsag opened 4 years ago

slimsag commented 4 years ago

Because of the way the Go memory allocator and gitserver work together, it uses up nearly all memory assigned to it regardless of the amount. This makes it almost impossible to alert on resulting in us removing the alerts for it.

The problem here is effectively that:

  1. RSS and VSS are very different topics, gitserver/Go use very little RSS but keep VSS around forever, basically.

image

  1. Go only cares about RSS, and treats VSS as effectively free - but in reality that is NOT true in k8s because cgroups OOM kills based on VSS.

  2. Go is not aware of VSS limits, like the ones we impose via k8s/Docker memory limits and does not appear intent on supporting that any time soon.

TODO: @slimsag resolve the issue in our Go services and/or write a blog post detailing/explaining the situation which has led to a lot of confusion elsewhere, including outside Sourcegraph.

github-actions[bot] commented 2 years ago

Heads up @davejrt @ggilmore @dan-mckean @caugustus-sourcegraph @stephanx - the "team/delivery" label was applied to this issue.