search

evanhempel / python-flamegraph

Statistical profiler which outputs in format suitable for FlameGraph (http://www.brendangregg.com/flamegraphs.html)
The Unlicense
395 stars 55 forks source link

Any docs on usage with wsgi apps? #1

Open andybak opened 9 years ago

andybak commented 9 years ago

i.e. how to get a log for an individual request?

evanhempel commented 9 years ago

So far its design is targeted toward profiling an entire application. You could run it against the entire WSGI app, but you'll probably find time waiting for a request would dominate the graph unless the server was extremely busy. You can work around that to some extent by filtering the output file to remove the stack frames that are waiting for a new request or filter to only include stack frames containing the method you're interested in.

grep -v waiting_method output.file > removed_waiting.out
or
grep function_name output.file > filtered_on_request.out

I think I can add an API to allow only collecting information for particular methods, but I'll need to put some thought into how best to do that... I'll update here when I have a proposal.

cjgu commented 9 years ago

It would be really awesome to be able to use it as a decorator to profile individual functions.

evanhempel commented 9 years ago

@andybak I've added code to allow filtering: python -m flamegraph -f function_name mywebapp-start.py should now work for profiling just the functions of interest. Let me know if this satisfies your usecase.

evanhempel commented 9 years ago

@cjgu I saw you made a few commits for prototype decorator support. How did that work? It seems like a useful feature. I have a concern with the code I saw so far that if we start and stop the thread for each function invocation it could end up being high overhead (on the other hand threads are lightweight so perhaps not). I'm interested to hear your thoughts.

cjgu commented 9 years ago

@evanhempel I tried it out on a flask app, decorating the entry point for a specific request handler, testing one request only by running curl against the endpoint. I was a bit disappointed in the frequency of the sampling however, getting out around 10 samples over a 500 ms period that the request took without sampling. From what I understand this setup should only start/stop the thread once right?

evanhempel commented 9 years ago

Strange. With an interval of 0.001s and a function runtime of 500ms if the profiling code had no overhead and the threads ran perfectly in parallel then we'd expect 500 samples. Now those ideal world assumptions are obviously not true, but I'd still expect more than 10. This needs some investigation, the reason for so few samples could be that the profiling code takes significant time, or that the context switching between threads is a comparatively large number of seconds...

I'm sure we can get those numbers better, but even so the flamegraph profiling method isn't geared toward short timeframes...

evanhempel commented 8 years ago

FYI, I stumbled across a python flamegraph project aimed Django, which maybe you'll find useful: https://github.com/23andme/djdt-flamegraph

I did notice that its using a different method to capture stack frames (signal module instead of using threads). It may be that that is a better profiling method, if so I'd consider switching this project to use that method as well. I haven't done any testing with that method yet, if you try it I'd be curious to hear what you find...