Heatmap creator hangs if called with another profilers

[soupault]

I'm trying to profile http://scikit-image.org/docs/dev/auto_examples/filters/plot_inpaint.html . vprof {ch/hc/mh/hm} plot_inpaint.py works units of seconds, while vprof cmh ... doesn't finish even within 10mins.

[soupault]

UPD: vprof cmh ... completes the workers for c and m, but hangs on h.

[nvdv]

Seems that it's caused by matplotlib and multiprocessing (like this one http://stackoverflow.com/questions/16254191/python-rpy2-and-matplotlib-conflict-when-using-multiprocessing) It works for me if I completely disable plotting. UPD: Works If I run heatmap calculation in the same process as vprof.

[soupault]

Not sure... I agree that It works when plt.show() is commented or substituted with plt.savefig(...). But it also works when plt.show() is presented for any combination of 2 profilers.

Hmm..., it also hangs on RuntimeProfile if executed as vprof hmc .... I'd say there could be something wrong with the handling of the 3rd profiler process.

[nvdv]

Sorry for any confusion I introduced. Currently runtime profile and heatmap calculation run in separate process that's forked from main process. If heatmap calculation runs in main process, everything works fine (you should edit code to do that).

[nvdv]

Now heatmap calculation runs in same process as vprof does. It fixes current issue for me on OS X. Can you confirm that issue is gone (in case if you're using dev version)?

[soupault]

@nvdv As I said eariler, the issue seems not to be related to Heatmap itself, but rather to the 3rd profiler called. Below is the result on the git master.

[user@machine restoration]$ vprof cmh plot_inpaint.py --debug
Running RuntimeProfile...
Running MemoryProfile...
Running CodeHeatmapProfile...
Starting HTTP server...
Created new window in existing browser session.
127.0.0.1 - - [24/Feb/2016 22:06:20] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [24/Feb/2016 22:06:20] "GET /vprof.css HTTP/1.1" 200 -
127.0.0.1 - - [24/Feb/2016 22:06:20] "GET /vprof_min.js HTTP/1.1" 200 -
127.0.0.1 - - [24/Feb/2016 22:06:20] "GET /favicon.ico HTTP/1.1" 200 -
^C

Opens (corrupted):

---

At the same time,

[user@machine restoration]$ vprof hmc plot_inpaint.py --debug
Running CodeHeatmapProfile...
Running MemoryProfile...
Running RuntimeProfile...

hangs.

[nvdv]

Can't reproduce it on Linux (I've used Ubuntu). Could you share more details about your setup/environment? Thanks!

[soupault]

Hmm... I've manually installed extra dependencies: phantomjs, mock, pylint, ... before casting make install. The issue with corrupted page has gone. Now I'm able to get the expected result for vprof cmh plot_inpaint.py.

Latest example (hmc) is still hanging. I bet you need something like https://github.com/nvdv/vprof/commit/c8866a5858ac65fb567b06783701875fcb689e59 for other profilers.

[nvdv]

There's shortcut to install all necessary dependencies for building vprof from sources (still requires npm): make deps_install Memory profiler already works in same process as vprof does. Do you have any issues with c option alone?

[soupault]

@nvdv neither with c nor with ch/cm.

[nvdv]

cProfile runs in single process in singleprocess branch now. Can you test it? Thanks!

[soupault]

@nvdv, It doesn't hang now for any combination of c, m, h. Although, the Flame chart/RuntimeProfile results aren't being rendered when c isn't the first specified profiler. In other words,

• c, ch, cm, cmh, chm - works perfectly;
• hc, mc, mch, hcm, mhc, hmc - renders well Heatmap + Memory, but doesn't render Flame chat (there is a corresponding tab in the generated page, but it is all gray).

[nvdv]

Thank you! It appears that issue with empty tab is a GUI issue and unrelated to the current one.

[soupault]

@nvdv looking forward for the final version! :) Thanks to you as well, very nice attention!

[nvdv]

Should be fine now. Pushed recent updates to singleprocess branch.

[soupault]

@nvdv I confirm, Flame chart is being rendered fine now. Also, the tabs are always sorted now, but it is rather an advantage.

Nothing to add from my side. Thanks for the great work!

[nvdv]

Thanks for your contribution! On Mar 1, 2016 00:08, "Egor Panfilov" notifications@github.com wrote:

@nvdv https://github.com/nvdv I confirm, Flame chart is being rendered fine now. Also, the tabs are always sorted now, but it is rather an advantage.

Nothing to add from my side. Thanks for the great work!

— Reply to this email directly or view it on GitHub https://github.com/nvdv/vprof/issues/26#issuecomment-190395681.