search

aternosorg / modbot

An open source moderation bot with advanced features developed by Aternos.
MIT License
127 stars 61 forks source link

Adjustment of default safeSearch likelihood according to Google recommendations and address current overzealous behaviour #587

Closed Aresiel closed 1 year ago

Aresiel commented 1 year ago

This PR is made in response to ModBot being noticeably overzealous in its detection and automatic removal of inappropriate images. Investigation revealed that the default likelihood threshold is set to 1 (corresponding to LIKELY).

Google states

In situations where you want to avoid false positives and flag images that you are very sure contain pornographic content, you would remove only images that the API identifies as VERY_LIKELY. If you aren’t as concerned with false positives and want to be on the safe side, you would flag and review all images that return POSSIBLE and above. Ultimately, your use case will determine how you handle the classifier’s response. (Sara Robinson, Filtering inappropriate content with the Cloud Vision API)

Stating that for all labels, including and above POSSIBLE, you would flag and review images. While reserving automatic removals for VERY_LIKELY in order to avoid false positives.

This change would not only better with the, albeit-not-absolute, recommendations set forth by Robinson but most importantly address the currently-present overzealous behaviour.

Possible partial invalidity: The discussed behaviour has been observed in the Aternos Discord server, and works under the assumption that the settings for that guild have not been modified to further lower the threshold.

CLAassistant commented 1 year ago

CLA assistant check
All committers have signed the CLA.

JulianVennen commented 1 year ago

Possible partial invalidity: The discussed behaviour has been observed in the Aternos Discord server, and works under the assumption that the settings for that guild have not been modified to further lower the threshold.

Are you trying to discuss what the value should be on the Aternos discord, or what the default value should be in general?

Aresiel commented 1 year ago

Possible partial invalidity: The discussed behaviour has been observed in the Aternos Discord server, and works under the assumption that the settings for that guild have not been modified to further lower the threshold.

Are you trying to discuss what the value should be on the Aternos discord, or what the default value should be in general?

In general, but my experience which led to me looking into this, stems from the Discord, and I can't be sure what threshold is used there so I am assuming it is the default value.

This PR regards the general case.

I have also brought it up to the Aternos Discord server because I don't believe their case necessitates the lower threshold either. Although I presume it is possible.

JulianVennen commented 1 year ago

We're not going to change the option on the Aternos discord.

Aresiel commented 1 year ago

We're not going to change the option on the Aternos discord.

This PR is with regard to the general case, I don't believe the general Discord server requires the lower threshold, in support of which I reference the Google Cloud blog post discussing the usage.

JulianVennen commented 1 year ago

The general discord server doesn't even have access to this feature

Aresiel commented 1 year ago

The general discord server doesn't even have access to this feature

I'd still argue that default settings should fit the general case, see for example in the case of self-hosting (or this feature being enabled further).

JulianVennen commented 1 year ago

This feature is not going to become "enabled further".

If you self-host ModBot and enable this feature, you probably want it to be more agressive.