softdevteam / warmup_experiment

Experiment designed to investigate JIT warmup times.
Other
8 stars 2 forks source link

Make clear the timings of segments #288

Closed ltratt closed 7 years ago

ltratt commented 7 years ago

At the moment we often don't really know what the segments on plots are, because the y-axis is so zoomed out that we can't see what's going on. This means that we sometimes appear to have a flat line but with changepoints, which can be hard to understand and defend.

I have a couple of vague ideas for this. First, on the plot itself, we could (above each segment) print out its mean (and variance?). Second, we could plot an inset for the final changepoint in a plot (or, if it's a slowdown, maybe the changepoint that causes that?).

snim2 commented 7 years ago

Ah yes, I hadn't forgotten this, but it isn't altogether simple. The problem with annotating segment means in the plots is that it is possible to have very short segments, which would not leave enough space to show any annotated text. In most cases this is not true of the last segment (but it might be true in a no steady state benchmark).

Adding some more data in the insets sounds more plausible, but that will make them larger, so there may be more charts where we can't show the inset at all.

ltratt commented 7 years ago

If a segment's really small and inconsequential, I guess that not showing the times is irrelevant. Maybe we just need to show the "important" segment's timings?

ltratt commented 7 years ago

OK, so I think there are probably two aspects to this issue:

1) Finding the important changepoint/changepoints: there's no point showing the user information that isn't going to affect their understanding of a plot. For flat, there are no interesting changepoints (easy!). For warmup/slowdown, there might be one or two interesting changepoints: if the steady-state segment has a mean "much" different than all other segments, then just showing that is fine; if the mean of that segment is similar to another segment, then showing both of them is probably necessary to understanding them.

2) Working out how to display this information to the user. We could do a second inset (but do we sometimes need a third? or, for "interesting" cases, maybe we drop the current "first" inset?). Or we can add timing information (means? means and variance?) directly onto the plot. Or ...?

snim2 commented 7 years ago

This was something we used for our down debugging and testing, and then removed from the repo. I think we can safely close this issue, as we did not need the requested feature "in production".