Scanning and reporting of illegal material

[ProbablePrime]

Is your feature request related to a problem? Please describe.

This issue is a little heavy, so read on with caution.

We need to put in some checks for illegal material into Resonite. All User generated content platforms should ideally do this as it protects victims, bystanders(who bump into the content) and Resonite itself from this content.

These checks should:

1. Figure out if material is illegal
2. Block the storage of this material on Resonite
3. Report the user who uploaded said material to Resonite
4. Report the user to the relevant authorities. (usually National Center for Missing & Exploited Children(NCMEC) or local equivalent.

Describe the solution you'd like

Microsoft provides a dedicated service to do this: https://www.microsoft.com/en-us/photodna It is free for many businesses our size. I have already opened up a dialogue with that team to discuss.

It provides an API that,

1. Takes an image
2. Hashes it
3. Compares it to a list of known illicit hashes
4. Reports the category of the llicit material if needed
5. Provides an optional API to report directly to NCMEC

We'd add that to our regular asset processing to cover this area.

Describe alternatives you've considered

Users in this thread have provided alternatives. I will work on a full plan.

Additional Context

Some of you may be concerned about false positives, to that:

• This is not AI
• This uses Image hashes derived from complex hashing functions
• False positives using hashes are possible but very very very unlikely.

I'm making this a public issue because its good to be transparent about our plans here. I've also previously implemented this exact setup while working at Microsoft so I'm very familiar with it.

EDIT: Some of you are worried about implications still, and that's fine. I've altered some wording to help, but remember that this is just a GH issue and not an implementation. A full plan will be thoroughly researched and shared when I can. That's the next thing you'll see on this issue. Not a release.

Requesters

Beethoven via security ticket.

[epicEaston197]

To be clear is this feature meant to detect adult content or is it meant to detect content that is illegal and should never have been uploaded to the platform to begin with?

[ProbablePrime]

Illegal only.

[SlyTheFloof]

Illegal only good

[JackTheFoxOtter]

I don't know the error rates of this, but comparing against known hashes should be relatively tight. Regardless, I do hope this won't be a fully automatic system and still require manual review in case something flags the system, maybe not before preventing the asset upload, but definitely before contacting any authorities.

[Azavit]

This is a big deal...

For the protection of the people on the platform, the protection of YDMS and the protection of the people in the images. This is very much an ASAP feature.

[kcseb]

While I agree that there should be some form of reporting and handling of CSAM and similar content... I believe that PhotoDNA is absolutely the wrong way to go about this. It's notorious for false-positives in large scale deployments, namely Discord as a great example.

Additionally, reporting directly to authorities is a terrible idea. There should be a full procedure for this that fully revokes the abusing users ability to use Resonite and any services related to it before it gets to the extent of going to the authorities. Plus, with it automatically going to authorities, any time there is a false positive or error, that's automatically passed on which is terrible if then Resonite is flagged on their end for abuse due to so many reports that aren't even really warranted.

I get it, its great you guys want to combat CSAM or whatnot, but personally, I do not think that this is at all the way to do it.

[PointerOffset]

While I agree that there should be some form of reporting and handling of CSAM and similar content... I believe that PhotoDNA is absolutely the wrong way to go about this. It's notorious for false-positives in large scale deployments, namely Discord as a great example.

Do you have any sources to cite on this? If the tool is actually that ineffective it'd be important to see documentation of that.

[epicEaston197]

PhotoDNA is absolutely the wrong way to go about this. It's notorious for false-positives in large scale deployments, namely Discord as a great example.

Yeah discord users got banned and reported to authorities because they uploaded an image of a guy eating popcorn

[ko-tengu]

Discord's problems are mostly their own homebrew AI image detector that has a billion and one false positives, and it doesn't have to go directly automatically to authorities and very few platforms use that part of the API.

Having this or some very similar service is absolutely a necessity for any platform that allows uploading arbitrary images, and having the human element of double checking potential matches helps eliminate the false positive issues.

[ProfessorDey]

Yeah I'm very much against this being a fully automated system for any and all active measures. There has been more than enough evidence of false positives ruining people's lives through automated systems that any such matching system really does need human oversight to manually trigger further measures only after things have been confirmed. As much as I understand the mental health implications for the person dealing with such flagged content, that's ultimately necessary if you want to not screw over innocent people. Beyond that there's also the issue of relying on an external commercial US based source to tell you what's illegal or not, especially for a business not based in the US (Not so much for only CSAM, which is relatively universal in the west but you only state 'illegal' which covers a hell of a lot more depending where you are).

Just from the FAQ there are some points of concern about the impact integrating this would have: "Customers based outside the USA will need to self-determine other reporting requirements based on local law." So this API doesn't actually help, you still need to file all the paperwork/reports manually for the country the service is actually based in, not NCMEC. "customers of the PhotoDNA Cloud Service authorize Microsoft to take steps to monitor and audit their usage of the PhotoDNA Cloud Service. Customers authorize Microsoft to provide aggregate reports to NCMEC that summarize the number of images (matched to signatures of known [CSAM] images) a customer uploaded on to the PhotoDNA Cloud Service." Meaning even internal human measures would need extensive documentation at each step of verifying something is not illegal and false positives may still be actively pursued regardless by NCMEC even when the relevant authority for the country Resonite is based in is notified of a false positive, meaning you'll at minimum have to deal with two different authorities on the subject.

While I understand the intent behind these sorts of measures being implemented, they have a huge chance of going catastrophically wrong if just thrown in with no oversight and a lack of thorough protocols being laid out for countermeasures against false positives hurting people unfairly.

[ProbablePrime]

I think we need to get real here, If you wanna a claim about it having false positives I'm going to need evidence in the form of a primary source and not associated with Discord, who uses other tools ontop of or in addition to this. We had 0 false positives for my entire time at Mixer. If you have bad things to say about this system that you've heard. I need a source. Otherwise I'm going to ignore it.

I also want to clarify that we aren't just going to spring this on you. Even getting access to the system takes months.

A full plan will be made and communicated here but it will NOT be sabotaged by a rumor mill.

I understand the issues with other systems and automated reporting but this is based on hashes alone and therefore the false positive chance is based on how strong the hash is and not on how good AI is at looking at images. There's no AI in place here.

If primary sources are supplied, further research on the matter will be done.

Please standby for a full plan, once I have a contact with the PhotoDNA team.

EDIT: Im also hiding off-topic comments, that feed the rumor mill. Users can still see these messages if they show them. This isn't me censoring you, this is me requiring a primary source for a claim.

[kcseb]

Based on these constraints, the PhotoDNA perceptual hash should be reversible to a recognizable image. Although multiple viable results are likely, all should be visually similar.

PhotoDNA does not detect flips, mirroring, 90-degree rotations, or inverting. However, it is supposed to detect visually similar pictures. Digitally alter less than 2% of the picture in very specific locations can effectively avoid detection. Moreover, these edits can be applied to non-salient regions of the picture.

Someone who wants to generate false-positive results only needs to modify a few selective portions of the picture. Forcing false-positive results can be used to justify plausible deniability in a court of law. (If you're involved in a CP/CSAM case, make sure your attorney receives the picture and not just a claim that the hash matched. If the evidence doesn't have the same SHA1, SHA256, or other strong cryptographic checksum, or isn't visually similar as identified by a human, then have the evidence excluded. It's not that I'm pro-CP, it's that I've heard of cases where people were accused based only on hash matches.) From: https://www.hackerfactor.com/blog/index.php?archives/931-PhotoDNA-and-Limitations.html

The aim of this paper is to give a first overview of the basic behavior of PhotoDNA with respect to robustness and its false positives. It can be stated that the hash provides very good results, however, it also does not provide the extreme performances that are sometimes mentioned in the public discussion. Some properties, such as the value distribution of the 144 hash elements, can be considered in more detail in future work. Likewise, it would be worthwhile to find a more efficient solution for the hash comparison.

As a more high level conclusion, PhotoDNA is a solid robust hash algorithm with potential of improvement. It needs to be stressed that the algorithm is not specifically desgined to deal with CSAM but will work with any other content as well. Infrastructures for content identification established for identifying CSAM therefore can be misused for other purposes by simply replacing the hash data base for comparison. From: https://dl.acm.org/doi/fullHtml/10.1145/3600160.3605048

A popular example that Louis Rossman covered. https://gizmodo.com/google-csam-photodna-1849440471

This article covers Apple's use of PhotoDNA but still illustrates the point.

Perceptual hashes are messy. The simple fact that image data is reduced to a small number of bits leads to collisions and therefore false positives. When such algorithms are used to detect criminal activities, especially at Apple scale, many innocent people can potentially face serious problems. From: https://rentafounder.com/the-problem-with-perceptual-hashes/

PhotoDNA is there, and is proven to work, but has also been proven to have flaws leading to serious issues not only due to its method of work (simple hash matching) but simply due to the lack of human involvement. Automation is great in many places, but when you're dealing with delicate content like CSAM, there unfortunately has to be a human, somewhere in that chain, verifying and making cheques & balances to ensure everything is done correctly and handled in a way that if need be, when law enforcement is involved, its for a legitimate reason, not because PhotoDNA flagged something that just so happened to hash match, but still be a harmless bit of media.

[ProbablePrime]

Oh great, thanks for your sources! I will add them to my research and plan work!!

I'll also be seeking to meet with the PhotoDNA team to bring anyone's concerns directly to them.

[ProfessorDey]

You'd have to define primary source given that everything about these systems, both Apple's and Microsoft's, are hidden behind NDAs such that no access to the code or real statistics is available. My objections are given to the implementation of such systems in general, not just one specific product. Obviously there's the Apple case with their clientside scanning, but in terms of research into how these work, I'd point out these as points of research, particularly the work of one Dr. Neal Krawetz, as mentioned here: https://www.osnews.com/story/133803/one-bad-apple/ and supplied by the man himself here: https://www.hackerfactor.com/blog/index.php?archives/931-PhotoDNA-and-Limitations.html

In regards to other solutions such as 'Safer', this ycombinator thread involves a developer mentioning that their benchmark for false positives is only as low a 1/1000: https://news.ycombinator.com/item?id=21445452

Edit: Looks like kcseb already mentioned some of the same research

[ProbablePrime]

I define primary source, as exactly what you've provided huge amounts of information I can read, process and factor into a plan from people who know stuff!. Thank you.

[epicEaston197]

I do have a question though is this entire system useless if the user doesn't have an Internet connection? They could upload it completely locally and host sessions by direct IP only or even lan servers

[ProbablePrime]

Yes LAN sessions would not be covered by this. While that's really bad, there's nothing we can do about that. There's nothing stopping a user from having material locally if they dont send it elsewhere. But this is also true with basically any other platform.

[ProfessorDey]

Also including links from the bottom of the second page I linked since they're easily missed: "Update: It only took a few months for Anish Athalye to implement an AI system that reverses PhotoDNA hashes back into pictures. He has a detailed writeup (with working code) and I have my own followup blog entry." Which naturally raises all sorts of additional potential privacy concerns, especially if PhotoDNA hashes are calculated for each uploaded image in the cloud.

[ProbablePrime]

Eesh, Okies. Thank you for the sprinkled alternatives too.

[ko-tengu]

I guess since it's off topic; I'll restate that you should absolutely do this but make sure to have a human element double checking reports.

[ProbablePrime]

Your statement related to Discord, which is PhotoDNA plus other stuff with no source. This new one is about automation and so that's totally fine! Thanks for the feedback!

[ProbablePrime]

I made some adjustments to wording in a few places, but once again to be clear. This is an idea that needs research to turn into a plan and your feedback is part of that process. Thank you.

[coolymike]

I'll add to this some concerns that come up whenever other platforms start to implement such checks:

First off, a human needs to be involved. The "low false positive rate" is for single images. When dealing with collections of images, like photos on someone's phone, or textures and thumbnails of Resonite objects, these checks are done very often, and the chance of false positives goes up quickly.

A human being involved also avoids the second issue, where some images out of context appear innocent, but are flagged as CSAM by automated systems. For example, a frame of a CSAM video, where the frame itself doesn't contain anything illegal. Some solutions for detecting CSAM (like Apples system) have vulnerabilities where a hash collision can be generated with any 2 arbitrary images. This would allow targeting a user and having them flagged by the automated system. To avoid this causing damage, a human person needs to verify the content of the images.

Another issue is the potential for tracking the exact distribution of images between users, across platforms. If photoDNA gets (the hashes of) these images, it's trivial to compare them and track where they were distributed, and between who. This can be abused for government censorship, for example when anti-governmental propaganda is shared among users.

However, as Resonite is not a private platform by any means (SignalR as messaging service, the asset variant system, etc), the risk of tracking is already there, and the platform should not be used for private matters. Altough, giving a service like photoDNA all (hashes of) images can centralize this type of data, making things easier for a government-size entity to track people using content uploaded to Resonite.

[ProbablePrime]

I'm now in contact with the Internet Watch Foundation as a referral from the PhotoDNA team. I'll be discussing IWF's tools and possible membership with them over the next few weeks.