Define procedures for moderation workflow

[soledadli]

Summary

This issue aims at:

• collecting feedback on moderation flow
• defining procedures for moderation workflow so that we can mock a detailed wireframe for moderators.

Please comment on this issue all questions you think will need to be discussed for the moderation workflow to be created successfully.

Details

Questions (Discussed & Solved)

• How many moderators’ approval need to be achieved for publishing experiences?
• Can moderators see other moderators' decisions (approval, disapproval) or do they remain anonymous in the platform?
• Can moderators message each other on the platform?
• If users‘ experiences are disapproved, can they ask for a new pair of moderators for a new decision?
• If new submitted experiences get disapproved again, does it count as the second warning or still count as one?
• If experiences are disapproved, is it mandatory for moderators to comment on why they are disapproved or shall moderators give reasons out once users ask for them?
• What are the selection criteria for selecting new moderators?
  • Are current moderators responsible for selecting new moderators? If so, how many approvals should a user receive to become a new moderator?

Moderation Workflow

Updated Version (Dec 13th, 2021)

Original Version

Related Issues

• https://github.com/alan-turing-institute/AutSPACEs/issues/267
• https://github.com/alan-turing-institute/AutisticaCitizenScience/issues/566

Suggestions for tasks that can fix this issue

• [x] Discussion in the AutSPACEs meet-up
• [x] Suggestion on the moderation workflow

Who can help

Anyone interested in moderation.

[KirstieJane]

Really great prompts for the discussion. Susanna and @JamesSCTJ may have some thoughts that they've aleady put into this.

Here are some of my thoughts with justifications to get you started (I'm sorry I can't be at the meet up as I have a full day meeting for a different project!)

• How many moderators’ approval need to be achieved for publishing experiences?
  • KW: For now, lets go with 1 moderator. I think we're going to struggle with volunteers anyway and we don't need to overcomplicate. You can always keep me as a "final decider" if something needs to be escallated.
• Can moderators see other moderators' decisions (approval, disapproval) or do they remain anonymous in the platform?
  • KW: They'll need to see IF there is a decision, and you're right that somewhere in the background we'll need an audit trail so we can follow back who has made a decision BUT I think I'd prefer for the names to be only available to administrators rather than to all the moderators. Seems like too much personal information to be sharing for no great benefit.
• Can moderators message each other on the platform?
  • KW: A firm no on my side on this question for two reasons. 1) We don't need the complexity of the infrastructure. If people need to communicate with each other they can do that using another platform (email, private issue trackers etc). 2) This is part of the reason why I say we need just one moderator per experience and 2/3 "final deciders". Moderators should not be getting into discussions on the platform with each other. We need training sessions and "continuous reflection" sessions to allow the moderators to share their perspectives, but as community and governance events rather than as part of the functionality of the platform.
  • KW: If a moderator isn't confident in their moderation decision - they need a flag to say "please can someone else come and have a look". But we don't need the messaging to happen in the platform, and I'd recommend that the escallation goes to a small number of administrators rather than to another community moderator (as it is likely a hard decision!)
• If users‘ experiences are disapproved, can they ask for a new pair of moderators for a new decision?
  • KW: This is a great question and I think Susanna and James have thought about it a lot so I'll wait to hear their perspectives. I think we do need some sort of appeals process... but I don't like the idea of people being able to send back their submission over and over until a moderator says yes! So it needs to go to the project administrators for a final decision, not round and round!
  • KW: Following along from that - we don't really need a huge amount of functionality around changing the moderation decisions. They could be overwritten in the database by the project administrators, but ideally we would train moderators to flag if they are unsure before making a decision rather than having to change in the future.
• If new submitted experiences get disapproved again, does it count as the second warning or still count as one?
  • KW: This is a really important clarification in the process. If people are engaging with the feedback in a respectful way then I don't think getting the experience sent back is an automatic warning. The warnings should only come in to play when the submitter has been rude to a moderator or is refusing to accept a decision. This needs updating in the policy, and links with my perspectives above that the project administrators' decision is final (given a reasonable appeals process).
  • KW: The platform itself doesn't need the functionality to track this back and forth - these incidents are likely to be sufficiently low occurance that they can be managed by a dedicated email or private issue tracker.
• If experiences are disapproved, is it mandatory for moderators to comment on why they are disapproved or shall moderators give reasons out once users ask for them?
  • KW: Again, I think S & J have thought about this a lot - my memory is that we DO want to give feedback at all times, probably best to build this into the workflow.
• What are the selection criteria for selecting new moderators?
  • Are current moderators responsible for selecting new moderators? If so, how many approvals should a user receive to become a new moderator?
  • KW: We aren't there yet.... but we also don't need this functionality in the platform so not a blocker!
  • (To be honest, we don't have criteria for selecting.... any moderators! Nevermind new ones! Or I guess we're all new at some point 😉)
  • KW: Just for clarity - I think we're a long way off actually running the platform with real data... so while I think this is good to think about / brainstorm - I don't think we need to worry too much about it. Happy to incorporate thoughts that members at the meet up group have though!

[GeorgiaHCA]

How many moderators’ approval need to be achieved for publishing experiences? 1

[GeorgiaHCA]

Can moderators see other moderators' decisions (approval, disapproval) or do they remain anonymous in the platform? Moderators remain anonymous to each other, but moderation decisions are visible. Admins can see each moderator's decisions and can send them a private email if there is a problem

[GeorgiaHCA]

Can moderators message each other on the platform? No, they will not have that option on the platform. There may be a separate Slack channel for moderators to talk to each other outside of the platform.

[GeorgiaHCA]

If users‘ experiences are disapproved, can they ask for a new pair of moderators for a new decision? No, this won't be necessary.

[GeorgiaHCA]

If new submitted experiences get disapproved again, does it count as the second warning or still count as one? We will hold off on the warnings for now

[GeorgiaHCA]

If experiences are disapproved, is it mandatory for moderators to comment on why they are disapproved or shall moderators give reasons out once users ask for them? Moderators will give a comment in a text box with an explanation telling you where the issue is/where to correct and referring to the code of conduct

[GeorgiaHCA]

What are the selection criteria for selecting new moderators? Moderators will have to be familiar with the code of conduct and must be able to apply them (via. practice cases); we will aim to have a significant number of autistic moderators

[soledadli]

Thanks for all the inputs and discussion! I will close this issue now.

[KirstieJane]

@soledadli - this issue should close when the content has been captured in the github repository - I have reopened. Please can you follow up with notes in a pull request?

[soledadli]

@KirstieJane Got it. I will capture the notes in the Github repository and later close the issue then.

[soledadli]

I have the following questions about reporting the harmful reports:

• Who can submit harmful reports? Every viewer or registered user?
• What is the mechanism for the harmful report?
  • Can viewers/registered users label triggering content for published experiences on their own if they think moderators may miss some points?
  • Do they need to file a report and let moderators decide how to re-label the triggering contents?
  • (We discussed in the meet-up last time that even though harmful reports are made, experiences will remain in the platform. )

[soledadli]

Meet up from November 30, 2021:

Harmful reports will be sent to moderators. Administrators will have the access to view all the harmful reports, but will not receive the notifications per experience.

[soledadli]

Moderators will select experiences on their own for moderation. Once experiences are selected, they should moderate within 24 hours. Otherwise, the experiences will be put back into the waiting pool.

[soledadli]

Email should be avoided in the platform as it is not user-friendly and can be quite hard for document the process.

[soledadli]

Let me know if I miss anything from the meet-up discussion today. @GeorgiaHCA @skfantoni @badgermind @jhlink @anoura12

[soledadli]

I have the following questions about reporting the harmful reports:

• Who can submit harmful reports? Every viewer or registered user?

• What is the mechanism for the harmful report?

  • Can viewers/registered users label triggering content for published experiences on their own if they think moderators may miss some points?
  • Do they need to file a report and let moderators decide how to re-label the triggering contents?
  • (We discussed in the meet-up last time that even though harmful reports are made, experiences will remain in the platform. )

We will hold off the "flag for harmful reports" feature for MVP as it will complicate the current infrastructure. Apart from that, further clarity will be needed for the "harmful reports" process.

[soledadli]

The role of "administrators/reviewers" need to be further discussed, as the "flag for help" feature will add workload for the development side.

[soledadli]

I am wondering whether 24-hours may be too short for moderation. Shall we provide an option to extend moderation time or generally extend it to a longer time? @GeorgiaHCA @skfantoni @badgermind

• SL: A feature to extend the time may complicate the infrastructure. I am wondering whether we can extend the time.

[jhlink]

I am wondering whether 24-hours may be too short for moderation. Shall we provide an option to extend moderation time or generally extend it to a longer time? @GeorgiaHCA @skfantoni @badgermind

• SL: A feature to extend the time may complicate the infrastructure. I am wondering whether we can extend the time.

For the time being, let's try 24 hours. If moderators provide feedback that it's too short, then we can increase the time.

[soledadli]

I am wondering whether 24-hours may be too short for moderation. Shall we provide an option to extend moderation time or generally extend it to a longer time? @GeorgiaHCA @skfantoni @badgermind

• SL: A feature to extend the time may complicate the infrastructure. I am wondering whether we can extend the time.

For the time being, let's try 24 hours. If moderators provide feedback that it's too short, then we can increase the time.

@jhlink Got it! I will keep the 24 hours limit then.

[soledadli]

Thanks James @jhlink for providing such detailed feedback on moderation workflow on Slack. Here is the documentation of the feedback:

Harmful Reports Workflow

• What does this mean? Published Experiences write harmful reports . It’s not clear to me what harmful reports entail or what function they’re supposed to fill when moderating reported experiences.
• As for modifying the code of conduct, I don’t think it’s related to moderating a reported experience. I think it’s an independent feature because it is not functionally required when reporting/moderating a reported experience.

  Selecting Experiences Workflow

• I think New Experiences SQL is a little misleading. I think it’s clearer to say Pending Pool of New Experiences or a collection of experiences pending review. Submitted experiences are submitted and stored already on the SQL database. Moderators selecting or unselecting them only modifies the experience in that it’s being reviewed by a moderator.

  New Experiences Workflow

• I’m a little confused here. Is this process describing the process of moderating ambiguous experiences rather than new experiences? I think the name of the process should be more specific.
• Lastly, if we’re avoiding emails, should we remove the mailto/mailfrom options ?

[soledadli]

@jhlink Hi James. Thanks again for the wonderful feedback! 🎉 Let me know whether the updated diagram and notes here clear out your confusion.

Harmful Reports Workflow

• What does this mean? Published Experiences write harmful reports . It’s not clear to me what harmful reports entail or what function they’re supposed to fill when moderating reported experiences.
  • Thanks for finding the mistake of the write harmful reports part. I updated the diagram. You can find it above.
  • I think the harmful report part is still under development. Would you mind sharing any insights about the harmful reports?@skfantoni @JamesSCTJ
• As for modifying the code of conduct, I don’t think it’s related to moderating a reported experience. I think it’s an independent feature because it is not functionally required when reporting/moderating a reported experience.
  • Modifying the code of conduct is not finalized. I think the moderation team is still discussing how the harmful reports work. I kept modifying the code of conduct in the workflow because we discussed in our last meeting that some harmful reports are issued because there is something missing in the code of conduct, and moderators may want to suggest adding the missing parts in the conduct. In this way, it is maybe handy if we have a button that directly leads moderators to the repository to raise issues.
  • The harmful report workflow has not been finalized.

    Selecting Experiences Workflow

• I think New Experiences SQL is a little misleading. I think it’s clearer to say Pending Pool of New Experiences or a collection of experiences pending review. Submitted experiences are submitted and stored already on the SQL database. Moderators selecting or unselecting them only modifies the experience in that it’s being reviewed by a moderator.
  • You are totally right! I changed the name to Pending Pool of New Experiences

    New Experiences Workflow

• I’m a little confused here. Is this process describing the process of moderating ambiguous experiences rather than new experiences? I think the name of the process should be more specific.
  • Ambiguous experience looks much clearer than unsure. I updated the name in the diagram.
  • I kept ambiguous experiences in the moderating new experiences loop as the ambiguous are new experiences at first. As moderators may not be sure about their decisions, these new experiences are flagged as ambiguous instead.
  • But I see what you mean about the process, as ambiguous experiences also appear in moderating disapproved experiences as well.
• Lastly, if we’re avoiding emails, should we remove the mailto/mailfrom options ?
  • We can discuss in this week's meet-up to see whether there are better options than mailing.

[soledadli]

Discussion notes from the meet-up on Dec. 9, 2021

• In the current MVP we don’t have a feature for contributors to re-add content. We want to avoid the issue of cross traffic when people go back and edit their contributions
  • • [ ] Could add in the instructions about editing comments post submission
  • • [ ] Could add a review feature where people can check their contributions before submitting
• A feature to add the reviewer role – understanding permissions required for this issue #204. Reviewers' roles have 2 major functions:
  • review ambiguous experiences and take positions for those requests
  • view moderator names and decisions to provide support in moderating experiences.
• Harmful reports:
  • moderator sets the labels for contributors and readers
  • labels would be visible to everyone once they’ve been set so readers can filter out triggering content