search

mozilla / addons

☂ Umbrella repository for Mozilla Addons ✨
Other
128 stars 41 forks source link

[Bug]: Many add-ons have been erroneously added to the review queue for overgrowth #15141

Closed wagnerand closed 4 weeks ago

wagnerand commented 1 month ago

What happened?

A lot of add-ons were recently added to the queue for growing over the average growth. This happened in error.

After an initial investigation, we believe this is happening because of an unanticipated negative hotness. That needs to be confirmed.

What did you expect to happen?

Only add-ons that really grew over the average should have been flagged.

Is there an existing issue for this?

┆Issue is synchronized with this Jira Task

diox commented 1 month ago

We also need to clear problematic NeedsHumanReview instances and call reset_due_date() on affected versions. That would happen automatically if the NeedsHumanReview are modified, but in this case we likely want those instances gone entirely. A migration can't easily do that because reset_due_date() method wouldn't be available, need to figure out if we could have a management command or admin tool that automatically resets due date for a given set of versions...

diox commented 1 month ago

After an initial investigation, we believe this is happening because of an unanticipated negative hotness. That needs to be confirmed.

This is definitely the case.

If the average hotness in a tier is -0.1, and the percentage set to 50 for that tier, with the current algorithm the threshold will be -0.0995. We likely want -0.05 in that case instead, since we want to consider increases in growth over the average.

diox commented 1 month ago

There is another issue with that algorithm: if the average hotness in a tier is 0.00, then multiplying by it won't do anything... We could work around that as well, but I think the true solution is going to be look at the Xth percentile instead.

diox commented 1 month ago

Command to fix the NHR/due dates: manage.py process_addons --task=delete_erroneously_added_overgrowth_needshumanreview --with-deleted

We successfully ran that in prod (and stage before that) today.

diox commented 4 weeks ago

The data has been fixed. We'll file another issue for the new algorithm.

wagnerand commented 4 weeks ago

The command above missed an edge case where a new version was submitted, causing the NHR flag to move from one version to another but changing the reason from "overgrowth" to "previous version had NHR set".

We identified and fixed those manually. The NHR flags for them have not been removed but marked inactive using the admin tools (NeedsHumanReview model page)