search

mozilla / mozanalysis

A library for Mozilla experiments analysis
https://mozilla.github.io/mozanalysis/
Mozilla Public License 2.0
10 stars 13 forks source link

Adds p_values for absolute and relative comparisons #198

Open danielkberry opened 10 months ago

danielkberry commented 10 months ago

This PR adds p-values to the output of mozanalysis.bayesian_stats.compare_branches. P-values are calculated using the p-value - confidence interval duality: given a bootstrapped confidence interval (as calculated using the existing methodology), this code calculates the corresponding p-value. A validation notebook is available here

codecov-commenter commented 10 months ago

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (c69380a) 76.31% compared to head (557fe3c) 76.83%. Report is 1 commits behind head on main.

Files Patch % Lines
src/mozanalysis/bayesian_stats/__init__.py 88.63% 5 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #198 +/- ## ========================================== + Coverage 76.31% 76.83% +0.52% ========================================== Files 23 23 Lines 992 1036 +44 ========================================== + Hits 757 796 +39 - Misses 235 240 +5 ``` | [Flag](https://app.codecov.io/gh/mozilla/mozanalysis/pull/198/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | Coverage Δ | | |---|---|---| | [py38](https://app.codecov.io/gh/mozilla/mozanalysis/pull/198/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | `76.83% <88.63%> (+0.52%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

danielkberry commented 10 months ago

One open question is around downstream impacts. @mikewilli, what changes need to be made to shared data models to accommodate the new field?