search

mozilla / missioncontrol

Real-time monitoring of Firefox release health
Mozilla Public License 2.0
21 stars 18 forks source link

Color coding to show whether “latest version” is better or worse #213

Closed rkothari19 closed 6 years ago

rkothari19 commented 6 years ago

Color coding to show whether “latest version” is better or worse than “previous version (avg)” https://data-missioncontrol.dev.mozaws.net/#/release/windows

wlach commented 6 years ago

We have to be careful about this, as a change in the median sample for any particular measure might not indicate a regression (since there are so many causes of natural variation for these numbers). But yes, for unambiguous cases we should do this (or something like it).

wlach commented 6 years ago

@mreid suggests maybe looking at cerberus for inspiration on how to detect that there is a significant, sustained change.

wlach commented 6 years ago

Right now we use color coding on the text in the main page to show significant (> 25% changes) in the mean for any particular measure relative to the previous release. I'm pretty happy with the way this turned out -- it gives a hint on things to look at without blazing alarm bells.

image

I'm going to close this for now. We may well want to tweak this further but I think we can open a new issues for that.