Open agentzh opened 9 years ago
Hmm, sorry, it seems that the most intuitive calculation should be more complicated than what I previously proposed. For that tiny example, maybe we should also "scale" the "after" result because it has way fewer samples in total and direct comparison between the original sample counts is like comparing oranges to apples :)
I think we can first scale
main;foo 1 2
main;foo;bar 0 4
into
main;foo 6 2
main;foo;bar 0 4
such that "before" and "after" results now have equal number of total samples and then perform the calculations:
foo: (6+0-(2+4))/(2+4) == 0 (0%)
bar: ((0-4)/4) == -1 (-100%)
These numbers look much more reasonable to me :)
Is there any update on @agentzh's request? It seems the interpretation from @agentzh sounds more reasonable to me.
Hi Brendan,
I'm very excited about the new differential flame graphs, especially the negated ones for assessing optimization results. Thank you for doing it! But I've found the delta percentage numbers rather confusing and are not what I'd expect.
For example, if a specific tower in the "before" flame graph disappears completely in the "after" graph, I'd expect the function frames in the negated graph show deltas like "-100%" but I'm getting ridiculously small numbers like "-0.8%" due to the way this delta percentage is calculated (which IMHO does not make much sense).
Please consider the following minimal example:
The resulting negated diff flame graph is shown here: http://agentzh.org/misc/flamegraph/tiny-negated-diff.svg
For these diff folded backtraces, the total reduction in
foo()
's time should be(1-(4+2))/(4+2) = -0.83
(or-83%
), but in the SVG, the "foo" frame shows-16.67%
. On the other hand, thebar
function should show(0-4)/4 = -1
(or-100%
), but the graph actually shows-66.67%
at thebar
frame. These numbers are quite far from my intuition.I can understand that the current calculation is the delta percentage regarding the whole sample space but for optimizations, I only concern about how radically a particular tower changes even though its overall ratio is not that big. And even for the whole sample space ratio calculation approach, the current results still do not look right ;)
What's your opinion on this?