search

golang / go

The Go programming language
https://go.dev
BSD 3-Clause "New" or "Revised" License
124.32k stars 17.7k forks source link

runtime/pprof,net/http/pprof: improve delta profiles efficiency and correctness #67942

Open korniltsev opened 5 months ago

korniltsev commented 5 months ago

Proposal Details

The issue

In Golang, allocation, mutex and block profiles are cumulative. They only grow over time and show allocations/blocks that happened since the beginning of the running program. Not only values grow, but the size of the profile itself grows as well. It could grow up to megabytes in size for long-running processes.

In many cases, it's more useful to see the differences between two points in time. You can use delta profile from net/http/pprof package. Using the delta profile requires passing seconds argument to the pprof endpoint query.

go tool pprof http://localhost:6060/debug/pprof/heap?seconds=30

What this does:

  1. Dump profile p0
  2. Sleep
  3. Dump profile p1
  4. Decompress and parse protobuf p0
  5. Decompress and parse protobuf p1
  6. Subtract p0 from p1
  7. Serialize protobuf and compress the result

The resulting profile is usually much smaller (p0 may be megabytes, while the compressed result is usually tens of kilobytes).

There are number of issues with this approach:

  1. Heap profile contains both allocation values and in-use values. In-use values are not cumulative. In-use values are corrupted by the subtraction. Note: It can be fixed if net/http/pprof package would use p0.ScaleN([]float64{-1,-1,0,0}), instead of p0.Scale(-1) for memory profiles - that would subtract allocation values and zero out in-use values in p0.
  2. It requires dumping two big profiles.
  3. It produces a lot of allocations putting pressure on GC.

DataDog's fastdelta

DataDog's fastdelta profiler uses another approach.

It improves the runtime/pprof approach by keeping a copy of the previous profile and subtracting the current profile from it. The fastdelta profiler uses a custom protobuf pprof parser that doesn't allocate as much memory. This approach is much more efficient, faster, and produces less presure on GC. It also doesn't require using two profiles. However, the fastdelta profiler still parses huge profiles up to megabytes, just to discard most of it.

Grafana's godeltaprof

godeltaprof does a similar job but slightly differently.

Delta computation happens before serializing any pprof files using runtime.MemprofileRecord and BlockProfileRecord. This way, huge profiles don't need to be parsed. The delta is computed on raw records, all zeros are rejected, and results are serialized and compressed.

The source code for godeltaprof is based (forked) on the original runtime/pprof package. godeltaprof is modified to include delta computation before serialization and to expose the new endpoints. godeltaprof relies on a bunch of golang runtime internal functions, specifically runtime_FrameStartLine, runtime_FrameSymbolName, runtime_expandFinalInlineFrame and runtime_cyclesPerSecond link and potentially even more internal functions . Relying on internal functions becomes harder and more dangerous due to https://github.com/golang/go/issues/67401.

Proposal

We propose to to allow efficient delta memory, mutex, block profiles collection, both in runtime/pprof for push-based integrations and net/http/pprof for scraping integrations.

The key points for improvements:

The specifics of API and implementation details are left to be determined during discussion of the issue after we see an agreement we need to address the issue and this is something that could be accepted into golang runtime

gabyhelp commented 5 months ago

Similar Issues

(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)

korniltsev commented 5 months ago

Oh, this is likely some sort of duplicate of #57765

prattmic commented 5 months ago

I don't believe this needs to be a proposal (no API changes, unless you think new runtime/pprof APIs are required to implement this?), so removing from the proposal process.

prattmic commented 5 months ago

Heap profile contains both allocation values and in-use values. In-use values are not cumulative. In-use values are corrupted by the subtraction. Note: It can be fixed if net/http/pprof package would use p0.ScaleN([]float64{-1,-1,0,0}), instead of p0.Scale(-1) for memory profiles - that would subtract allocation values and zero out in-use values in p0.

I recommend filing a separate issue for this, if one doesn't already exist. The rest of this issue is a performance optimization, but this is a real bug.

prattmic commented 5 months ago

cc @golang/runtime

rsc commented 5 months ago

I don't know, it seems like new API to me. (It's a new URL handler variant but that's still API.) Please do kick it over to the proposal process if it ends up being a non-trivial change.

prattmic commented 5 months ago

It's a new URL handler variant but that's still API.

Perhaps I am missing it, but I don't see what the new URL handler variant is. IIUC, the core of this issue is to make the existing /debug/pprof/heap?seconds=30 handler more efficient.

korniltsev commented 5 months ago

Perhaps I am missing it, but I don't see what the new URL handler variant is.

I had multiple ideas:

  1. Improve /debug/pprof/heap?seconds=30 (no knew API). This only fixes the issue for net/http/pprof package, but not runtime/pprof .
  2. Maybe a new delta implementation could live behind a runtime/pprof.Profile (new API - profile name, new URL ), this way it could solve issues for both net/http/pprof and runtime/pprof package. It would be nice to have an option to avoid sleeping for profile collection, just reuse data from previous profile collections.
korniltsev commented 5 months ago

I recommend filing a separate issue for this, if one doesn't already exist. The rest of this issue is a performance optimization, but this is a real bug.

I think there is an existing issue already https://github.com/golang/go/issues/57765 . It has no bug label thoughю