Metaculus / metaculus

https://www.metaculus.com
BSD 2-Clause "Simplified" License
54 stars 12 forks source link

Added account deactivation on spam detection of profile/bio updates #1484

Open CodexVeritas opened 17 hours ago

CodexVeritas commented 17 hours ago

Issue: https://github.com/Metaculus/metaculus/issues/1386 New spam detection flow:

Identification rates:

CodexVeritas commented 17 hours ago

@lsabor You mentioned that the 2-10% false positive rate was too high. Should I shoot for a 1-3%? There are some profiles which are hard to distinguish from spam. I'm guessing so few people change their bio that this is probably an ok rate (and people will understand why they were flagged when they are). Only 300 have bios and a comment/prediction of our 210,000 users (granted probably ~50% of those users are spam).

For reference, we technically have 210k users, 71k users with bios, 25k users have forecasted. Only ~300 have bios AND have forecasted/commented. Given this, I'm guessing that ~50% of users are spam. Over the lifetime of Metaculus this would have resulted in 6-30 false deactivations who could email us to remedy the situation (probably 50% from people doing something stupid like throwing html into their bio, or making something legit sound fishy). And actually this would be more like 0-10 since they might have changed their bios after the 7 day range

Though more iterations couldn't hurt, just curious how much of a difference it will make since they can reach out if there is a problem.

CodexVeritas commented 17 hours ago

Also looks like I need to apply flake8 formatting. Let me fix that real quick.

CodexVeritas commented 17 hours ago

@lsabor Who would I invite to request review? I should invite Sylvain for the final QA review because he assigned the ticket to me, but for the preliminary review, will just anyone from the team do?