runtime/pprof: no API to access deep stacktraces with MemProfileRecord and BlockProfileRecord

korniltsev commented 3 months ago

Go version

go version devel go1.23-b788e91bad Tue Jun 11 18:08:44 2024 +0000 linux/amd64

Output of `go env` in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/korniltsev/.cache/go-build'
GOENV='/home/korniltsev/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/korniltsev/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/korniltsev/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/home/korniltsev/github/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/home/korniltsev/github/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='devel go1.23-b788e91bad Tue Jun 11 18:08:44 2024 +0000'
GODEBUG=''
GOTELEMETRY='local'
GOTELEMETRYDIR='/home/korniltsev/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/korniltsev/github/go/src/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build1768278824=/tmp/go-build -gno-record-gcc-switches'

What did you do?

I compare stacks from runtime.MemProfile and runtime/pprof.WriteHeapProfile

https://go.dev/play/p/7iUI7DdMI3r?v=gotip

What did you see happen?

I see the MemprofileRecord stacks are truncated at 32 depth and pprof stacks has proper deep stacks.

What did you expect to see?

I expected MemprofileRecord.Stack() to return a slice with correct deep stack.

gabyhelp commented 3 months ago

Similar Issues

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

korniltsev commented 3 months ago

runtime.MemProfile and MemprofileRecord are used in [godeltaprof] (https://github.com/grafana/pyroscope-go/tree/main/godeltaprof)

It is probably possible to hack around the limitations by linking into runtime/pprof.memProfileInternal but in the light of https://github.com/golang/go/issues/67401 this now looks like less attractive idea.

felixge commented 3 months ago

I worked on the patch for adding deep stack traces. FWIW I initially tried to make deep stack traces available via the public API, see v35 of my CL.

The problem is that there are various go1compat challenges in doing this, so we'll need a proposal that either manages to solve the problem without compat issues, or build consensus for accepting a minor amount of compat issues. Specifically:

Are we okay with breaking apps that don't expect Stack() to return more than Stack0 frames?
Are we okay if the StackRecord types are no longer properly comparable? (in either the spec sense, or by containing a pointer that makes comparison meaningless)
Are we okay with changing the semantics of whether or not the slice returned by Stack() is backed by the Stack0 array in all cases?

korniltsev commented 3 months ago

One of the possible solutions (for godeltaprof) could be to have the same functionality in golang standard library and to deprecate the godeltaprof project alltogether.

https://github.com/golang/go/issues/57765 https://github.com/golang/go/issues/67942

znkr commented 3 months ago

We have a similar issue in with Google's internal peakheap profile implementation. As the name suggest, the peakheap profile keeps track of high-water-mark heap profiles. I was considering filing a proposal to add peakheap profiles to the runtime, but I didn't have enough time to think it through. Given that I am not the only one with such a problem, I am favoring the idea of exporting the larger stacks via a runtime API now.

Regarding the go1compat challenges: Do we have to change the existing API or could there be a v2 API (e.g. in a new package, say runtime/profile).

prattmic commented 3 months ago

cc @golang/runtime

rsc commented 3 months ago

I believe this is a duplicate of #43669.

mknyszek commented 3 months ago

@rsc I think there's some overlap, but this is really about a specific unresolved part of #43669. Meanwhile, #43669 is now mostly resolved, since as of https://go.dev/cl/572396, deep stacks are available everywhere except the runtime APIs mentioned in this issue. I think maybe we should close https://github.com/golang/go/issues/43669 in favor of this issue?

rsc commented 2 months ago

Sure, if you think #43669 is done, then feel free to close it.

Personally, I would say that the API to access the deep stack traces is runtime/pprof, and we just deprecate the old MemProfileRecord, BlockProfileRecord structs.

korniltsev commented 1 month ago

I assume we will have to link into new pprof_memPrifileInternal and co. if we want deep stacktraces? I am not missing anything, right?

golang / go