Add square map visualisation

[cdeil]

I had never seen the Sunburst visualisation before, and while it is very pretty, I don't find it very intuitive to browse. How much work would it be to add e.g. a square map visualisation?

[jiffyclub]

I'm personally of the opinion that the sunburst is a better visualization of this data than a square tree because things aren't piled on top of each other. The important thing to remember is that angular extent is the informative dimension.

In theory the two visualizations are related and amenable to the same input data. But developing new viz types is definitely going to take a back seat to improving performance.

[cdeil]

Probably the Sunburst just takes some getting used to (as would any other viz). I think #14 will help, too.

[embray]

I know when we first started on this I experimented with other visualizations, including square map, but I had a harder time making it look good.

[jiffyclub]

While working on the refactoring I came across icicle plots, which look like a close companion of the sunburst and so might not be terrible to add.

[cdeil]

I think I figured out why I find the square map more intuitive than then sunburst, it's because for me comparing areas easier than angles, i.e. it lets me more quickly identify which functions to look at.

Is this what the icicle plot does (area proportional to value of an item)?

[jiffyclub]

With the icicle plot it would be width, not area, so more similar to the sunburst's angular display.

[kburns]

+1 for the icicle plots, especially if they can be modified (from that demo) to maintain the same number of rows as the focus moves down the tree, which I think the sunburst does right now.

[jiffyclub]

Note that #53 adds icicle plots.

[scls19fr]

Tree map will be nice https://en.wikipedia.org/wiki/Treemapping

[scls19fr]

Some D3 examples https://bl.ocks.org/mbostock/4063582 (fixed)

a zoomable treemap (only one color) https://bost.ocks.org/mike/treemap/

http://www.billdwhite.com/wordpress/2012/12/16/d3-treemap-with-title-headers/

This one is nice also http://bl.ocks.org/ganeshv/6a8e9ada3ab7f2d88022

[cipri-tom]

On the topic of different visualisations, the flamegraphs are quite nice too. These are default for pyflame.

I haven't used icicle too much yet, but it seems the same idea, just upside down ? 🙃

[rwarren]

Squaremap/treemap would be excellent, and is what runsnakerun (py2 only) uses. See notes in #112 for reference.

Here is a D3 treemap implementation: https://github.com/d3/d3-hierarchy#treemap

[rwarren]

To highlight how a squaremap with with area representing cumulative call time can significantly help with profiling/optimization efforts, consider the example below.

(This is mostly just me capturing my notes trying to figure out how to get the info I need out of snakeviz, since runsnakerun is unfortunately dead)

This got waaaay too long, and is of questionable value to post, but I'm doing it anyway...

---

I have an API call that accesses a back-end database that I would like to optimize. Pure text profile outputs give these as the top 5 hitters from a tottime perspective (time spent in the function alone, not it's children):

1. 14% tottime --> pyscopg2.extensions.cursor.execute()
2. 6.3% tottime --> zlib.compress()
3. 4.5% tottime --> socket.setsockopt() (!?)
4. 2.8% tottime --> socket.sendall()
5. 2.5% tottime --> psycopg2.extensions.cursor.copy_from()

To really optimize this, I'd like to see a much larger fraction of time spent in those C extensions, since that's how you make "python" screaming fast. Looking at profiles from a tottime/bottom-up perspective like this is extremely helpful for many optimizations.

Let's look at the default 10-deep icicle chart for that profile:

Looking through that, it is impossible to find the top 5 functions by tottime. Considering only the 14% tottime cursor.execute() call, it would be nice if it jumped out at you, but it does not. Worse, no matter how much you dig you won't find it... because it isn't there!! The call tree is clipped to a depth of 10, and that cursor.execute() is deeper than that! The call tree display is top down, and we need the bottom layers in there.

Bumping the depth to 20 gets you this:

The top 5 tottime functions still do not jump out at you visually. After some code-aware exploration, I can find the cursor.execute() (highlighted in pink), but it doesn't seem like a big deal visually (aside from the 14% visible on hover after you find it). So much of the chart area is dedicated to call tree that I can't easily extract duration information, even looking at the bottom/leaf nodes. I also still can't find the other 4 of the top 5 tottime hitters in the visual, or what call branch they are invoked on.

The fundamental thing here is that icicle (and sunburst or flamegraph) visuals prioritize showing you a call tree rather than showing you what you often care about when profiling, which is identifying where all the time was spent during the profile. The call trees can be nice/useful to look at (although severely limited by python profiling limitations, per #112), but a treemap where area is based on call time would often be much more useful. kcachegrind and runsnakerun (py2 only 😢) both use treemap visuals for this reason.

Anther issue worth pointing out since those charts were pasted in, is that the displayed call trees can be extremely misleading (this is mainly discussed in #112). For example, in the 20-deep chart above roughly 1/3 of the area in the 20 deep icicle chart is completely bogus! This part of the visual is pretty much completely wrong and misleading:

This is because this particular profile has a call in it with significantly different branching, and it is called from many places. Sometimes it exits early, sometimes it does a lot. The lack of information on the cProfile output forces snakeviz to assume all calls branch in the same way, and it stamps out the same call tree for every single call. This would be true, even if said function exits early 99% of the time and takes 10 minutes 1% of the time. A squaremap based display (like runsnakerun) would technically have the same limitation with the call tree hierarchy, but since the primary display intent is portioning relative call-time, it is much less of a problem.

I don't have a squaremap comparison for the above example, but here is an old one showing a similar case (with psycopg2 and sqlalchemy) that demonstrates the prioritization on relative total time for a call, while preserving available call tree information:

The most obvious thing there is that you can see where most of the time is being spent at a low level.

---

In summary: A squaremap would be an awesome addition to snakeviz. 😄

[jiffyclub]

Heads up that I do have this at the top of my list of additions to snakeviz, but I think it requires updating D3 a couple versions and there have been some significant changes to the API that affect snakeviz, so I expect it to take a while to work through that. But hopefully we'll get there!