[Task]: Do not include DSA abuse reports in aggregates that have been decided on

wagnerand commented 2 weeks ago

Description

Abuse reports that apply to DSA and have been decided on by a human should not be included in the aggregate considerations.

Acceptance Criteria

  ### Milestones/checkpoints
  - [ ] Abuse reports that apply to DSA and have been decided on by a human should not be included in the aggregate considerations.

Checks

[X] If the issue is ready to work on, I have removed the "needs:info" label.
[X] If I have identified that the work is specific to a repository, I have removed "repository:addons-server" or "repository:addons-frontend"

┆Issue is synchronized with this Jira Task

eviljeff commented 2 weeks ago

Some clarifications @wagnerand:

Should reports be excluded them from the calculations only after they've been resolved with a decision? (the alternative is excluding all reports that should be manually reviewed - but may not have been at the time).
- The former would mean when we're backlogged with reports we don't miss problematic content; the latter is perhaps more accurate, because there would be no overlap between reports that are DSA relevant, and reports that are considered in aggregate.
If we're only considering reports with decisions, does any decision count, including non-decisions like AMO_IGNORE, for invalid/misfiled reports?
- (what about AMO_CLOSED_NO_ACTION, which is used when content is already disabled - although I'm not sure you would be reviewing those add-ons anyway?)

wagnerand commented 1 week ago

Some clarifications @wagnerand:

* Should reports be excluded them from the calculations only _after_ they've been resolved with a decision?  (the alternative is excluding all reports that _should_ be manually reviewed - but may not have been at the time).

  * The former would mean when we're backlogged with reports we don't miss problematic content; the latter is perhaps more accurate, because there would be no overlap between reports that are DSA relevant, and reports that are considered in aggregate.

We should do the latter. If we already ran into a backlog issue, surfacing more add-ons from aggregation won't help with the backlog, but will just increase the size of the queue and doesn't lead to those being handled any faster.

* If we're only considering reports with decisions, does any decision count, including non-decisions like `AMO_IGNORE`, for invalid/misfiled reports?

  * (what about `AMO_CLOSED_NO_ACTION`, which is used when content is already disabled - although I'm not sure you would be reviewing those add-ons anyway?)

In terms of moderator decisions, it's hard to make the perfect decision upfront. I'd lean towards including invalid/misfiled/non-actionable reports in the aggregation, but if we see a lot of add-ons being enqueued over and over again for this, we might have to reconsider that. Might be worth keeping that in mind for the implementation in case we need to change it later. Reports for content that is already disabled should not be included in the aggregation.

ioanarusiczki commented 8 hours ago

I tried understanding https://github.com/mozilla/addons/issues/9221 first, because I needed to establish the minimum number of reports that would make an addon be flagged.

There's not too many addons on dev with ADU but what I could find for an unlisted version is

https://reviewers.addons-dev.allizom.org/en-US/reviewers/review-unlisted/619960 -> I reported it once with version 3.0 installed (ADU is 5 so I think it should be reported once to be flagged)

I also reported once a listed version , the report is open and went to Cinder. (ADU is 9, so it should take one report to be flagged). I want to see if this is flagged after the cron job runs while the report is not being decided on (left open in Cinder).

mozilla / addons