face detection to speed up blur/paint process?

[nd-rw]

face-api would be quite easy to implement and it worked quite well on images of crowds (I used the SSD Mobilenet model with a minimum confidence of ~0.2). Once the detection function has been run over the image, you could just fill each bounding box (identified face area) with a blur or painted colour?

face-api does use the tensorflow-js library but from a quick analysis of the demo site it seems to be serving all the js code and the models from the github page, I'm not a privacy/security expert so up to you I guess!

There is also a nice little library called picojs which detects faces reasonably well in ~200 lines of pure javascript. I can submit a PR using either of these libraries, just let me know!

[everestpipkin]

I'll do a bit of research and see if they use non-local processing and how they run on phones. First priority is that its quick even on old mobile architecture, since a lot of this is happening in the field on older devices.

[nravic]

I've been working on something like this (re: scrubbing images and face detection) and can take this up if you'd like. I messed around with face_recognition (https://github.com/ageitgey/face_recognition) but found it kind of lacking in crowds. Maybe there's a better alternative.

[bMacSwigg]

IMO there's also value in keeping the code both simple and readable: it should be as easy as possible to verify that the tool isn't doing anything nefarious. If we add face detection, we should take extra caution, both because the code itself may be complex and because people may be more wary of a tool that is actively identifying faces.

OTOH, there's definitely value in this for ease of use. I've worked before with assisted facial recognition, which may be able to simplify the task. e.g., users could click once on a face, and the tool could automatically find the edges of that face and blur it.

[everestpipkin]

Actually, I think even a tap mode might be really helpful? Right now you have you actually move the touch/mouse for it to draw, and it registers mousedown as the start of that movement. But just being able to tap on a crowd of faces to blur a circle on each one would probably do a lot of the work here?

[everestpipkin]

I also did some testing on both of those libraries, but unfortunately they were catching the sky, the wall, a book, etc far more than my face, especially at a distance of more than 5 feet. Also, given the how facial recognition is often not written to perform correctly on people of color, I think its too much of a risk to integrate- at least with these tools.

[nd-rw]

Actually, I think even a tap mode might be really helpful? Right now you have you actually move the touch/mouse for it to draw, and it registers mousedown as the start of that movement. But just being able to tap on a crowd of faces to blur a circle on each one would probably do a lot of the work here?

I guess the challenge here is determining how big to make the circle? I agree that it would speed up the process quite a bit.

[raisingdibar]

For what it's worth, iOS has good native on-device ML frameworks that can take the majority of model filetypes, idk about Android's built-in support. So, provided that we accounted for bias when training, and things like masks, ML could still help. If the crossplatform frameworks could be integrated with the Vision framework on iOS (and Android's counterpart if any exists), that could be a route forward.

[everestpipkin]

Can you pass along these frameworks so I can do some testing?

I'm generally underimpressed with both the resources required for and the reliability of mobile face-tracking, especially in crowds that are wearing masks and given the historic problems ML has had operating on non-white skintones. So far, I've felt that its not responsible to offer an automation step that might not actually catch everyone, and that will significantly complicate the codebase. That said, if there is a readable solution that can run locally in browser and will operate effectively in these test cases I'm definitely willing to reassess.

[marcinciarka]

https://www.youtube.com/watch?v=WzqwmFRP5z0

@everestpipkin could this work? I'm having some trouble with the undo function, but otherwise it's working (well, definitely needs some threshold tweaking, cause guy's armpit was detected as a face).