mozilla / mozanalysis

A library for Mozilla experiments analysis
https://mozilla.github.io/mozanalysis/
Mozilla Public License 2.0
9 stars 13 forks source link

Faster decile inference #196

Closed danielkberry closed 9 months ago

danielkberry commented 9 months ago

This PR provides a new method for quantile inference using a "resampling-free bootstrap". It provides >100x speedups over the current implementation (comparison tests in the linked notebook).

This is valuable as, in some cases, the vast majority of Jetstream analysis duration is spent in the decile statistic.

codecov-commenter commented 9 months ago

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (20c13ee) 76.18% compared to head (ddca1c5) 76.31%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #196 +/- ## ========================================== + Coverage 76.18% 76.31% +0.12% ========================================== Files 23 23 Lines 974 992 +18 ========================================== + Hits 742 757 +15 - Misses 232 235 +3 ``` | [Flag](https://app.codecov.io/gh/mozilla/mozanalysis/pull/196/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | Coverage Δ | | |---|---|---| | [py38](https://app.codecov.io/gh/mozilla/mozanalysis/pull/196/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | `76.31% <83.33%> (+0.12%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#carryforward-flags-in-the-pull-request-comment) to find out more. | [Files](https://app.codecov.io/gh/mozilla/mozanalysis/pull/196?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla) | Coverage Δ | | |---|---|---| | [src/mozanalysis/frequentist\_stats/bootstrap.py](https://app.codecov.io/gh/mozilla/mozanalysis/pull/196?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=mozilla#diff-c3JjL21vemFuYWx5c2lzL2ZyZXF1ZW50aXN0X3N0YXRzL2Jvb3RzdHJhcC5weQ==) | `90.38% <83.33%> (-3.74%)` | :arrow_down: |

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.