Run images through an inappropriate content filter

[makyen]

Given that we see inappropriate content (porn, CSAM, etc.) posted as images, it would be good to run those images through an inappropriate content filter in order to detect such things. My expectation would be that an external service would be used, but there may also be packages available which process the image locally. Substantial investigation as to what's available, and what we might get free access to, would need to be done. We should at least investigate if there was a way for us to use the same service that SE is using.

[teward]

Note we have a legal issue here - simply having access to such content to generate hashes, etc. violates several US and other jurisdiction laws. Any system processing this would need an MOU from corresponding legal jurisdictions that we are operating such filtration and hash summing in accordance with legally defined rules and limits, thereby not violating federal CSAM or content laws. Additonally, any system running these has to be in a Terms of Use where this activity is permitted.

We need to deeply assess the legal security around this esp. if we intend to implement #11794

[makyen]

Hmmm... Legal issues were not ones which I had been considering, but I agree they could be considerable.

Would we have a legal issue if the only things we touch are the URL that's in the post and the response from the inappropriate image content filter that's performed by some third party (i.e., we pass the URL that we find in the post, not the image, to the third party; then the third party service is responsible for fetching the image from the URL)?

[teward]

(broken GH is broken, they deleted my message)

I believe that would resolve the issue, yes. I think most of the content filtering services have an MOU with law enforcement and work with law enforcement and notify on CSAM and stuff.

This would prevent local processing though because we then rely on a third party service.