Proactive rather than reactive TrollBot scanning

[bchrobot]

Is your feature request related to a problem? Please describe.

TrollBot scans after messages have already gone out, at which point the damage is done.

Describe the solution you'd like

TrollBot should scan messages before sending them and prevent flagged messages from going out.

Describe alternatives you've considered

N/A

Additional context

Future messages by the same user following a flagged message could also be flagged until an admin clears the user.

[bchrobot]

One possible implementation would be to add a 'FLAGGED' status to the options for message.send_status. Rather than sending the message, Spoke would just insert into the message table with FLAGGED. These messages could later be sent or dismissed (by setting send_stats = 'NOT ATTEMPTED') or "released" (actually sending it).

Recorded questions responses, tags, and/or opt-outs are not public-facing so there is no need to prevent/defer recording them at send time. The recorded data could still be reviewed for malicious tagging/categorization when the flagged message text is reviewed.

Thoughts? @ben-pr-p @kohlee @curtries

[ben-pr-p]

I think we should push using our trollbot webhooks to help our administrators react to trolls faster instead of implementing any automatic banning of users.

I think almost all trollbot alarms are false positives, which is ok, because sifting through 95% false positives is worth it to catch one troll.

However, that means that any automatic blocking / restricting behavior will force admins to take an extra action to resolve something, and 95% of the time, that will be unnecessary.

As a result, admins may be pressured to reduce their monitoring surface area, and end up not catching things that they should catch

[kohlee]

++ what Packer said.

A middle ground could be differentiating between troll alarms that show up in queue and troll alarms that prevent sending. Admins could note the severity level when creating a trigger word. Ideally, someone would choose to have messages that contain a slur be proactive but messages that contain something like a curse word or opposing candidate's name be reactive. In practice, an admin may choose to have ALL trigger words be high severity/proactive, and you'd run into the problem that Packer noted.