memory profiling and compute profiling the same code

[brianbruggeman]

I really like the output of this memory profiler.

However, I think that people who are interested in memory efficiency may also be interested in execution time. And in requiring the decorator (or some other memory profiling mechanism), we end up with code that requires modification each time we want to profile for both compute and memory. Finally, the memory profiler slows down execution time, so for a final product ship, we must remove the memory profiling mechanism. Making updates to code for just capturing profiling data is cumbersome and definitely not efficient.

Ideally, the memory profiler would require no updates to the code to perform and would function on the code in a similar manner as the cProfile module.

[brianbruggeman]

As a work around...

# Generic on-demand memory profiler
func = profile()(func)
func(*args, **kwds)

I wrapped the above in a docopt arg: ' -p --profile Runs memory profiler'

This way I don't need to have the decorator on the function and I can run my standard cpu benchmarking from the command line:

python -m cProfile script.py <script args>

[fabianp]

Thanks for the feedback.

Ideally, the memory profiler would require no updates to the code to perform and would function on the code in a similar manner as the cProfile module.

Indeed, I would love to see this implemented. I suppose the best way would be to be able to specify from the command line the function to be profiled, something like

python -m memory_profiler --func=my_module.my_func script.py

I think this would be doable but it would involve a bit of tweaking of the current codebase, as the trace mechanism sys.settrace would need to be activated whenever the function in which it enters matches my_module.my_func, that is, we would need to make a trace that is globally active and checks for every function call wether it corresponds to my_module.my_func.

[brianbruggeman]

I like this.

You might also be able to get away with setting up something like a specialized pdb and a callback on function enter/exit.