[Task]: Reviewer actions and user notifications inventory

Description

With DSA, there have been some changes to which party is informed when with what information. The reviewer tools indicate for many actions whether a notification email is sent or not, but it is not clear if this is still accurate. Also, it is unclear to reviewers if the review comments are included in the email or not.

Before making any changes, we want to get an inventory of the current state.

Acceptance Criteria

  ### Milestones/checkpoints
  - [ ] Collect a list of all reviewer actions (in the review tools) and their variations (e.g. resolving abuse reports, escalations, appeals or not), including the email template being used and whether the email contains the reviewer comments.

Checks

[X] If I have identified that the work is specific to a repository, I have removed "repository:addons-server" or "repository:addons-frontend"

┆Issue is synchronized with this Jira Task

@ioanarusiczki Is this something you or your team can help us with?

@wagnerand yes we will take care of QA work on this. Thank you

@wagnerand

That's still in progress but you can check the emails I attached in the urls below ( if you open an attachment from testrail, you must click full resolution to make content readable). I've done 2 examples for theme escalated because emails should be the same as for extensions reported for add-on policies , there is not point for me to go in circles. To be clear about scenarios I've wrote them here at pct. 1 (those apply everywhere).

addon is reported for add-on policies scenario 1,2,3,4

Scenario 1 https://mozilla.testrail.io/index.php?/cases/view/2766682 Scenario 1 https://mozilla.testrail.io/index.php?/cases/view/2766700 Scenario 2 https://mozilla.testrail.io/index.php?/cases/view/2766703 Scenario 3 https://mozilla.testrail.io/index.php?/cases/view/2766806 Scenario 4 https://mozilla.testrail.io/index.php?/cases/view/2778424

theme report from Cinder is escalated to amo Scenario 1 with approve https://mozilla.testrail.io/index.php?/cases/view/2778692 Scenario 3 with reject https://mozilla.testrail.io/index.php?/cases/view/2779017

I've noticed this afternoon that when content is disabled from Cinder, emails sent to authors no longer have the last paragraph with the appeals url -> this is only on -dev (on -stage it's still as before).

email sent for a theme disabled from Cinder on dev

theme disabled from Cinder on dev

email sent for a theme disabled from rev tools on dev

rejected from rev tools with appeal available

Thanks. I am not sure I follow all the scenarios. Andrew created an overview just last week, maybe this helps with the testing?

It's unclear to which high-level scenarios we are testing for each of the test scenarios. I have not looked at the actual emails yet.

Scenario 1 mozilla.testrail.io/index.php?/cases/view/2766682

This looks like we deny an abuse report. This can be either done via Resolve jobs -> Approve or with one of the dedicated approval actions. The appeal covers Deny, but not Grant?

Scenario 1 mozilla.testrail.io/index.php?/cases/view/2766700

Looks like abuse-report deny because not enough info or developer feedback. This looks clear.

Scenario 2 mozilla.testrail.io/index.php?/cases/view/2766703

This appears to be the Grant variant of scenario 1, with a force-disable action. What about a rejection of versions? This also seems to be about a developer appeal, which is granted using force-enable, perhaps based on the moderation action on the back of the appeal in part 1 of this scenario? We should also include a case for rejections/reinstating of versions.

Scenario 3 mozilla.testrail.io/index.php?/cases/view/2766806

This looks like accepting an abuse report and choosing Rejections (mentioned in scenario 2). The appeal case of the author is unclear to me. What is the relation to the new version? The new version can be ok or not ok, and also the appeal for the old version(s) can be granted or denied independently.

Scenario 4 mozilla.testrail.io/index.php?/cases/view/2778424

How is this different two the first part of scenario 2? Is it just about denying the appeal here, where it was a grant in 2?

I'll stop here, all in all it's very difficult for me to understand the scenarios, to see if we covered everything and what the expected emails should be compared to the ones we are actually sending. Can we expand the scenario description and create an overview of all the different scenarios? Thank you.

I tried gathering some examples focusing on the emails sent and the comments done by reviewers in the past 2 weeks, that was the main focus. Ofc there could be many other aspects to focus on when it comes to abuse reports testing.

I looked at the guidance , this will help for a few actions that I did not try or focus too much along the year such as approve multiple versions and confirm approval. What I tested with was Approve (as approve an addon version) and the Approve from Resolve reports section (yeah, confusing a bit, there are many ways to get the same email). Making checks on reporting a guid for an unlisted version not installed I see no problems so far and same emails are sent with what's already in my examples above.

My main focus since January was on listed approved versions. The unlisted versions were later resolved as how it should behave https://github.com/mozilla/addons/issues/14993 https://github.com/mozilla/addons/issues/14985

The only thing I do not understand is at Developer Appeal Grant -> I never unrejected versions first before approving a new version (not sure why that is required). However, granting developer's appeal with Approve is sending the correct emails. When Approving a new version while an appeal is present in the DSA section, it means that previously another version has been rejected (based on a report sent by someone). So this would be "scenario 3" from what Mathieu wrote a while ago in https://github.com/mozilla/addons/issues/9387 "Content taken down, author appeals, we change the decision and bring the content back up." This can also be done with force disable/force enable (if the entire flow goes in rev tools). Appeals can also be forwarded from Cinder and if that's the case the reviewer has the possibility to force-enable it back from rev tools ( because cinder disables content, it does not have a reject like rev tools).

It's unclear to which high-level scenarios we are testing for each of the test scenarios. I have not looked at the actual emails yet.

If by high level scenarios you mean https://github.com/mozilla/addons/issues/15074 then this has been quite recently introduced and it only covers the Cinder reports. We're at https://github.com/mozilla/addons/issues/15085 for now (in progress).

This appears to be the Grant variant of scenario 1, with a force-disable action. What about a rejection of versions?

The answer is yes, this scenario can be done with reject or force disable from rev tools but I did not have the time these weeks to try so many scenarios and make notes. Reason why I used reject at scenario 3 which is how scenario 2 would continue when the developer sends the appeal and decision taken by the reviewer is to reinstate content.

Scenario 2 Content not taken down, reporter appeals, we change the decision and disable. The content goes down, the author can appeal (the only time an appeal decision can be appealed) From there, in that second appeal we can change the decision to (Neither the reporter nor the author can appeal that final decision):

Approve (reinstating the content)
Confirm the decision to disable

This looks like accepting an abuse report and choosing Rejections (mentioned in scenario 2). The appeal case of the author is unclear to me. What is the relation to the new version? The new version can be ok or not ok, and also the appeal for the old version(s) can be granted or denied independently.

All I know (from qa point of view) is that techicaly when Reject is used to take content down then you need a new version for approval to reinstate content. Has been done with https://github.com/mozilla/addons/issues/9496 after discussions that unreject wouldn't work https://github.com/mozilla/addons/issues/9471.

How is this different two the first part of scenario 2? Is it just about denying the appeal here, where it was a grant in 2?

It is not, the difference is that scenario 2 includes that the report is first denied (with approve actions, as already described at scenario1), reporter sends the appeal and the second time it's granted -> this will generate a new email which is sent to the reporter (that the initial decision was incorrect), this email is not present with scenarios 1,3 or 4. At Scenario 3 or 4 ( which is Content taken down, author appeals, we confirm the decision, the equivalent of Developer Appeal Deny from the doc) the content is taken down based directly on the report and not on a following reporter appeal.

What I try to say is that going through these scenarios even if I did not use all possible reviewer actions with each, the emails are pretty much what I've already added in these test cases. I think you should take a look, there is a small difference in content between versions rejected versus addon disabled (the emails sent to authors when content is disabled) which is expected, I've verified this a while ago and repeatedly all year round when doing regression testing. Meanwhile I'll try to get used to the new doc's terminology, I might be using the one from https://github.com/mozilla/addons/issues/9387 which I got used to, so in case I confuse you please ask me to clarify on slack.

mozilla / addons