search

weaveworks / scope

Monitoring, visualisation & management for Docker & Kubernetes
https://www.weave.works/oss/scope/
Apache License 2.0
5.86k stars 714 forks source link

Simplify in-field profiling #983

Open 2opremio opened 8 years ago

2opremio commented 8 years ago

Right now, to get profiling information of the probe/app you one needs to install golang. On top of that, in the case of the probe, one needs to restart the probe with --probe.http.listen.

We should make this simpler. We have missed a couple of chances of getting detailed performance reports from users because it's too complicated.

See: https://botbot.me/freenode/weavenetwork/2016-02-17/?msg=60310921&page=1 and https://botbot.me/freenode/weavenetwork/2016-02-18/?msg=60361117&page=1

For instance we could dump a cpu/memory profile when receiving a signal.

tomwilkie commented 8 years ago

I would love a scope profile command...

2opremio commented 8 years ago

To improve the situation and remove the in-field go tool pprof dependency, I've tried to get the profiles directly with curl but I've miserably failed:

$ curl http://172.16.0.3:4040/debug/pprof/profile?seconds=30 > profile.raw
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1072  100  1072    0     0     35      0  0:00:30  0:00:30 --:--:--   288
$ go tool pprof profile.raw
Entering interactive mode (type "help" for commands)
(pprof) top
70ms of 70ms total (  100%)
      flat  flat%   sum%        cum   cum%
      70ms   100%   100%       70ms   100%  
(pprof)

(Note how the resulting profile is useless)

Maybe somebody knows better and can tell me what I'm during wrong.

2opremio commented 8 years ago

Maybe somebody knows better and can tell me what I'm during wrong.

It was a simple as providing the binary as an argument to pprof!

The problem is that curl doesn't symbolize the output profile: https://groups.google.com/forum/#!topic/golang-nuts/-Y1NQhexJmU

I will add this to the instructions in the README.

For completeness: go tool pprof obtains the profile (in the same way as curl) and then symbolizes each address using the remote pprof/symbol endpoint, see https://github.com/golang/go/blob/master/src/cmd/pprof/internal/symbolz/symbolz.go

bboreham commented 5 years ago

Note the current docs for obtaining a profile are at https://github.com/weaveworks/scope/blob/da2cce36912c1bd720178f3c3f6cbafd8a174bf1/site/building.md#profiling

and it's not necessary to run go to collect the profile - as long as you have the same binary locally the tool will figure it out. You can just curl xxx:/debug/pprof/profile