runtime: mechanism for monitoring heap size

[bradfitz]

Tracking bug for some way for applications to monitor memory usage, apply backpressure, stay within limits, etc.

Related previous issues: #5049, #14162

[aclements]

@robaho, if you're on Linux (or probably most BSD kernels) and enable core dumps, you should get a core dump. You can then analyze that core's heap using viewcore (which definitely still has rough edges, but can answer a lot of questions).

[aclements]

The intent is to get some experience with that API and make sure it actually solves problems (and doesn't create new ones :). We're planning to roll it out as an experimental API within Google and I was also hoping to get some adventurous open source users to try it out (I should email golang-dev), but neither of these has happened yet.

Just a quick update on this. We've had several projects trying out SetMaxHeap. It helps, but this experimentation has uncovered a few rough interactions with other parts of the runtime. The biggest problem is that large object fragmentation can cause the heap's RSS to grow significantly larger than the allocated size of the heap (#14045). As a result, for systems that suffer from large object fragmentation, there needs to be a large (and somewhat unpredictable) buffer between the reserved memory and the max heap. The other known problem is that stacks and globals don't currently count toward the GC trigger, but do count toward the RSS, and since stacks change dynamically it's hard to account for this overhead when setting the heap limit (#19839 and #23044). Both of these issues also cause problems for other reasons (wasting memory and failing to amortize GC costs). I've prioritized fixing both for Go 1.12.

[andreimatei]

@aclements thank you for your interest in this area. For what it's worth, all this is a pretty big deal for CockroachDB, who's trying to do memory accounting and would like to trust the runtime to stay within given limits.

[vitalyisaev2]

I have an urgent need in a tool that helps to understand what's actually going in a Go process address space, why RSS keeps growing despite of memory limitations and so on. Also I need to use cgo, which makes problem even more complicated. Currently I have to use a set of tools like pprof, valgring --tool=massif, viewcore (introduced few weeks ago), and some self-developed tools. But it looks like I can see only different aspects of the problem, not an entire problem.

For example, I see that process has 5GB RSS. Go runtime says that it takes 2GB (though only 5% of them is used, and other 95% are idle). cgo library says that it uses < 500MB in it's caches and other internal data structures. And no one knows what consumed the remaining 2.5GB. I can only speculate if it's Go runtime cost (for instance, because I have profiler enabled as well), or it's a memory leak due to cgo (malloc without free).

[aclements]

@vitalyisaev2, please open a new issue or send an email to golang-nuts@googlegroups.com. In it, please elaborate on what you mean by "Go runtime says that it takes 2GB". The runtime exports many different statistics, and it's important to know which one you're talking about. I would start by looking closely at all of the runtime.MemStats statistics, since that should tell you if it's on the Go side or the C side. If it's on the Go side, viewcore is probably the right tool to find the problem. Please also describe the time-scale of the problem, since some things (like returning memory to the OS) happen on the scale of minutes.

[rsc]

@aclements, what do you think the decision is here that NeedsDecision refers to? Is there a different bug for your memory pressure work? Should this issue be closed?

[fortuna]

For the record, there's a change to improve how memory is restored to the OS that is scheduled to be released on Go 1.12 for Linux and Go 1.13 on iOS, preventing OOM: https://github.com/golang/go/issues/29844.

[rsc]

Closing as a duplicate of #29696.

[gopherbot]

Change https://golang.org/cl/227767 mentions this issue: [release-branch.go1.14] runtime/debug: add SetMaxHeap API