Open jenniferplusplus opened 2 months ago
This is great, I think this is exactly the kind of thing I was hoping for.
I wanted to go into the interviews and recruiting with some common understanding of what questions we're trying to answer with the research, and how we would find those answers. And also, at least for myself, some specific strategies that are worth pursuing during the interviews.
Here are some high-level questions I'm hoping we can answer through this research:
What additional context can we help bring in for individual moderation actions? What's essential context versus what can/should be easy to reach? In short, how can we make it so no one ever has to connect to the database and ask questions via SQL? Are there things we can try to pull in from other instances via their APIs?
What long-term context can we help preserve across moderation actions for the long term? Mastodon has notes, and while I'm not entirely sure what all you can attach notes to, would formalizing certain things into enums/flags such as this user is often targeted for hate be useful? Is there a need to go beyond "notes" in providing formalized context that can be understood at a glance, instead of reading through text?
What sort of automated systems can we help setup for proactive, low-hanging fruit? For example, disallowing signups based on patterns in usernames, ignoring reports from user accounts with no visible history, anything you might otherwise think "I could do this with a shell script but it's a PITA".
What are some specific things we can do to help admins coordinate moderation decisions across instances, and what would implementing that safely look like? What prior art exists in this area? Is there an opportunity to define standards to help these efforts across other instance software such as gotosocial? Like, having a way to coordinate / subscribe to blocklists, but also allow your instance to define overrides or whatnot to ensure certain things never break, or reviewing incoming changes from a subscribed resource and accepting/rejecting individual parts.
How much mistake-proofing and/or auditing are we going to need to bundle in to specific things? What can't be "undone" because it interacts with the outside world? What actions are moderators most afraid of messing up and losing trust or regretting?
How can we present lists of things for review in a way that lightens the load a bit? Are there ways to determine what needs the most attention or has higher severity? For example, would it be helpful to group open reports in the list by reporter/reportee?
I'm thinking about more specific questions to ask that will help get at these topics, and will post them separately.
This seems great!
One thing I've considered a few times is whether reports are actually the correct abstraction to center for moderator actions. Does it make more sense to organize around something like cases? Something that can encapsulate reports, involved parties, notes, actions taken, and maybe other things? (Are there other things? Is that the right set of nouns?) How often would that framework be more effort than it's worth?
Does it make more sense to organize around something like cases? Something that can encapsulate reports, involved parties, notes, actions taken, and maybe other things?
I like this; it occurs to me as a natural abstraction, one that could be made mostly invisible, would allow for easier linking/cross-referencing between cases and their subject matter (users, notes, etc) and could become polymorphic to encompass thing initiated by items other than reports.
Here are some questions I think would be good to ask moderators in an interview:
a few more questions for the research findings after today's first (wildly successful) interview:
And another I'd like to explore:
When I interview people for the purpose of building tools to help their work, I am typically trying to learn five things:
The latter two questions are I feel fairly direct and straightforward; at least when I am talking to professionals, they're often able to articulate answers with a minimum of coaxing and it's possible to learn a lot without dancing around the point.
The first three questions tend to be the opposite. Few people are introspective enough to be able to give straightforward answers to these questions without a lot of coaxing and guidance.
I don't have a template for these sorts of interviews: my approach as someone with ADHD is to a) be present, and b) to improvise. Everyone is different; they think about their work differently, they'll have different ideas about what's important, what's easy or not, weigh consequences differently, and even use different domain language to talk about their work. Additionally, the more distant someone is from the front-line of the work, the less they typically understand the nuances of the problems involved or the potential for consequences from a mistake to create catastrophe, but sometimes the opposite is true: a manager might have a much deeper understanding of the consequences of mistakes than any of the front-line workers.
Generally my approach is to start by asking them to describe their typical work session. In our case, this is, it's time to do some moderation work. How do they know this needs to be done? What are they thinking about in the space between "oh it's time to put on the moderator hat" and finding out why they need to? If there are multiple things to do, how do they decide what to do first? How do they evaluate claims, make decisions? There will typically be a lot of opportunity to sidestep into particulars. This is also a good point to find out what's frustrating about the workflow, what works well, and hopefully ask about how they know if they've made a good decision.
After this initial pass, and particularly as I proceed through talking with multiple people, I'll typically ask what advice they might give someone new to doing this type of work: what are the important things to watch out for, what are the places they need to be extra careful? As I learn more about the domain from previous conversations, I'll sometimes propose a scenario and potential approach to later participants – this is partly to calibrate how well my understanding is coming along, but also partly to see how their approaches differ. Mostly this is to learn about their mental models and attempt to break into the unspoken assumptions about domain data.
I'll typically close by describing some approaches to problems I've been thinking about and ask for their feedback. In this case, I might mention that I've been thinking about improving the defederation interface, you enter an instance's url and it tells you how many people this would impact: how many people on your instance would be impacted? How many people on the target instance do people on your instance interact with? I might show them a sketch or mockup of such an interface – this is less about validating the idea and more about listening to their response. What does this idea bring up for them? It's another good opportunity to calibrate my understanding of the problem so far and discover the unspoken domain data assumptions.