mozilla / addons

☂ Umbrella repository for Mozilla Addons ✨
Other
127 stars 41 forks source link

[Task]: Make growth threshold before flagging relative instead of absolute #15049

Closed diox closed 3 weeks ago

diox commented 1 month ago

Description

We currently flag add-ons that have their number of users grow above a certain rate threshold depending on their usage tier, but that threshold is an absolute number that is fixed for a given usage tier.

That approach doesn't play well with seasonality, we sometimes have higher or lower growth rates at specific times of the year for various reasons, and we don't want to constantly adjust the thresholds.

To address this, we should make that threshold relative to the average growth (called hotness internally) for add-ons in the given usage tier instead of absolute.

Acceptance Criteria

  ### Milestones/checkpoints
  - [ ] Growth threshold before flagging is relative to the average growth in the usage tier instead of absolute

Checks

┆Issue is synchronized with this Jira Task

diox commented 1 month ago

Relevant code: https://github.com/mozilla/addons-server/blob/b9af959f76f9669de41fd4aca7c6d219557e5baf/src/olympia/addons/tasks.py#L528

For example, if an Addon has average_daily_users set to 500, it then belongs to the UsageTier that has lower_adu_threshold=100 and upper_adu_threshold=1000. If the growth_threshold_before_flagging on that UsageTier is 5, then instead of flagging the add-on when their hotness value is 0.05 or above (hotness is a percentage), we'd flag it if it's 1.05 * {average hotness value for all add-ons in that usage tier} or above.

Finding the add-ons should still be doable in a single query, though we probably need separate queries to determine the mean hotness for each usage tier.

wagnerand commented 1 month ago

Note on timing: To avoid running into the issues again that triggered this work item, this would need to be done for the last push this year, at the latest.

ioanarusiczki commented 3 weeks ago

On AMO dev: these are the add-ons which belong to Tier2, growth threshold set to 5%, value is 0.378 -> the add-ons with 0.6 and 0.8 should be flagged

https://reviewers.addons-dev.allizom.org/en-US/reviewers/review-listed/633520 hotness 0.1 https://reviewers.addons-dev.allizom.org/en-US/reviewers/review-listed/633521 hotness 0.3 https://reviewers.addons-dev.allizom.org/en-US/reviewers/review-unlisted/633522 hotness 0.6 https://reviewers.addons-dev.allizom.org/en-US/reviewers/review-listed/633523 hotness 0.8 https://reviewers.addons-dev.allizom.org/en-US/reviewers/review/616918 hotness 0.00

-> checking the review pages the auto-approved versions for 633522 and 633523 got flagged