JuliaPerf / PProf.jl

Export Julia profiles to the pprof format
MIT License
154 stars 17 forks source link

add progress bar to allocation preparation #53

Closed IanButterworth closed 2 years ago

IanButterworth commented 2 years ago

I ran the allocation profiler with sample_rate=0.1 and the process of preparing the pprof took >10 minutes with no output before I quit.

This adds a progress meter to the preparation stage.

There's still a long silence before the warning message shows, but I guess that's a Profile issue.

julia> PProf.Allocs.pprof(from_c = false)
┌ Warning: The allocation profiler is not fully implemented, and may have missed some of the allocs. For more info see https://github.com/JuliaLang/julia/issues/43688
└ @ Profile.Allocs ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.8/Profile/src/Allocs.jl:134
Analyzing 9239305 allocation samples...  1%|▉                                                                                                                       |  ETA: 0:48:08

cc. @NHDaly @vilterp

codecov-commenter commented 2 years ago

Codecov Report

Merging #53 (c52920e) into master (18d842d) will not change coverage. The diff coverage is 0.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #53   +/-   ##
=======================================
  Coverage   72.81%   72.81%           
=======================================
  Files           3        3           
  Lines         298      298           
=======================================
  Hits          217      217           
  Misses         81       81           
Impacted Files Coverage Δ
src/Allocs.jl 0.00% <0.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 18d842d...c52920e. Read the comment docs.

vilterp commented 2 years ago

Reading this again, I realized that a significant amount of time is spent in Profile.Allocs.fetch() which won't be captured by this progress bar 🤔

I.e. out of two stages — decode and pprof — this is only capturing pprof. Kind of makes me want to bring back https://github.com/JuliaLang/julia/pull/43889, which deferred stack decoding so basically all the time would be in pprof 🤔

NHDaly commented 2 years ago

Yeah, @vilterp: I think that we should provide another API that only exports the raw data together with something like an lidict that users can use to decode the data offline, which will also allow for exporting the results to disk without needing PProf, which can be nice.

We talked about something like that in the past, i think.

IanButterworth commented 2 years ago

Note that Profile.fetch() was recently multithreaded https://github.com/JuliaLang/julia/pull/43816 Maybe something similar is possible?

vilterp commented 2 years ago

Yes, I think we should be able to do something similar :)

Also, for what it's worth, the strategy there of partitioning a collection and spawning a thread to process each partition seems like a good addition to the standard library (i.e. parallel_map or something), so both profilers (and other stuff) could use it — not sure where such a function would live 🤷‍♂️

IanButterworth commented 2 years ago

Exactly.. https://github.com/JuliaLang/julia/pull/43919