[X] I have checked the repository for duplicate issues.
What enhancement would you like to see?
Currently Chubby relies on nsfwjs for content detection. While this model is good, it has been known to present a LOT of false positives. So much so that the deployment we currently uses basically disables this feature entirely (the threshold is extremely high)
To be fair, the version of the library/model we use is pretty old, however at the end of the day it is still a community driven model. We may find better success in more robust models. Examples would be:
The private detector model by Bumble (which the Duolicious model is based on)
These are models in use by large scale companies, today. So I believe they can be trusted? Testing would need to be done however.
We also should move away from automatically marking a user as having sent NSFW content and instead flag them for review by moderators. This will, of course, slow down the response time of actual NSFW content being removed, but it will lessen the amount of people being falsely punished (which is why we've basically disabled this feature in production)
wolfendale
commented
3 months ago
Checked Existing
What enhancement would you like to see?
Currently Chubby relies on nsfwjs for content detection. While this model is good, it has been known to present a LOT of false positives. So much so that the deployment we currently uses basically disables this feature entirely (the threshold is extremely high)
To be fair, the version of the library/model we use is pretty old, however at the end of the day it is still a community driven model. We may find better success in more robust models. Examples would be:
These are models in use by large scale companies, today. So I believe they can be trusted? Testing would need to be done however.
We also should move away from automatically marking a user as having sent NSFW content and instead flag them for review by moderators. This will, of course, slow down the response time of actual NSFW content being removed, but it will lessen the amount of people being falsely punished (which is why we've basically disabled this feature in production)
Any other details to share? (OPTIONAL)
No response